Software Engineer L5/L6 – Model Evaluations, Data Curation

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and construct a shared data curation framework, which includes reusable components, libraries, and workflows.

• Develop scalable LLM-driven data transformation pipelines utilizing raw sources such as the Netflix catalog and its metadata.

• Implement large-scale batch inference while focusing on quality and token cost efficiency.

• Formulate sampling strategies that ensure coverage, diversity, difficulty, and balance across various content and member segments.

• Create filtering and quality control techniques, which include LLM-as-judge, evaluation-model-based scoring, deduplication, and validation processes.

• Collaborate with researchers to evaluate the impact of curation decisions on model performance.

• Transform curated datasets into discoverable artifacts with versioning, explicit lineage, and reproducibility.

• Promote the adoption of shared curation practices within MEDC and partner modeling teams.

⛳️ Requirements

• Proficient in software engineering with Python, particularly in building reusable infrastructure, libraries, or frameworks utilized by fellow engineers and researchers.

• Experienced in developing LLM-driven data generation or transformation pipelines, including synthetic data, structured outputs, or large-scale batch inference.

• Practical experience with data quality methodologies, including sampling strategies, filtering, deduplication, and model-based quality scoring like LLM-as-judge.

• Comprehension of how data choices influence model behavior and the capability to design experiments that measure this impact.

• Familiarity with distributed data processing tools such as Spark, Ray, or their equivalents.

• Exceptional collaborative skills, especially when working with researchers, data scientists, and platform teams.

• Experience with LLM evaluation systems is essential for L6 candidates.

• Proven technical leadership in data and evaluation infrastructure; ability to establish technical direction for a multi-engineer initiative is crucial for L6 roles.

• Knowledge of dataset versioning, lineage, and artifact management practices.

• Skill in optimizing cost and throughput for large-scale LLM inference processes.

• Experience with human annotation workflows and aligning LLM judges with human raters.

• Familiarity with pipeline orchestration frameworks such as Metaflow, Airflow, or similar.

• Background in recommendation systems, personalization, search, or working with content catalogs and metadata.

🏝️ Benefits

• Health Plans

• Mental Health support

• 401(k) Retirement Plan with employer match

• Stock Option Program

• Disability Programs

• Health Savings and Flexible Spending Accounts

• Family-forming benefits

• Life and Serious Injury Benefits

• Paid leave of absence programs

• Full-time hourly employees accumulate 35 days annually for paid time off, applicable for vacation, holidays, and sick leave.

• Full-time salaried employees are granted immediate access to flexible time off.