Research Engineer – Pretraining Data

This is a fully remote position, open to applicants in California.

📋 Description

• Identify, assess, and gather important training data from web, coding, and technical resources.

• Create scalable pipelines that transform raw data into training-ready corpora through processes such as extraction, cleaning, deduplication, and filtering.

• Establish checks for data quality, coverage, and contamination, leveraging these to enhance collection and processing methods.

• Collaborate with researchers on data mixtures and sampling strategies for both pretraining and midtraining phases.

• Prepare and assess data for long-context learning, reasoning, and coding, testing synthetic data when applicable.

• Conduct controlled data experiments and utilize model evaluations to determine which sources, mixtures, and processing options are worth expanding.

• Ensure large-scale data processing is efficient, reproducible, and recoverable, with transparent source provenance and dataset versioning.

⛳️ Requirements

• Proficient programming skills in Python, with a proven track record of developing reliable data-processing or distributed systems.

• Experience in handling large datasets and navigating the practical trade-offs related to storage, computation, throughput, and reproducibility.

• Capability to diagnose data-quality issues and convert research objectives into quantifiable checks and experiments.

• Comfort in taking ownership of evolving technical challenges from start to finish and collaborating closely with researchers to test assumptions and enhance the system.

• Background in web crawling, document extraction, code-data collection, or extensive corpus curation.

• Familiarity with deduplication, text classification, quality scoring, synthetic data, or analyzing training-data contamination.

• Experience with data-mixture experiments, curriculum learning, or evaluating how changes in datasets influence model performance.

• Experience in preparing datasets for language-model pretraining, continued pretraining, or midtraining is a plus.

• Familiarity with distributed processing frameworks like Spark, Ray, or similar systems is advantageous.

• Experience with multilingual, mathematical, scientific, or repository-level code data is a bonus.

• Understanding of tokenization, training data loaders, and the requirements for large model-training runs is beneficial.

🏝️ Benefits

• Equity incentives.

• Flexible work arrangements, offering the option to work remotely or in-person at our offices in San Francisco.

• Visa sponsorship available.

• Relocation assistance for international candidates.

• Quarterly team off-sites.

• Opportunities for hackathons.

• Access to conferences and learning opportunities.