Research Engineer, QC Automation

This is a fully remote position, open to applicants in California.

📋 Description

• Develop systems to automate quality control for training data generated using HUD’s infrastructure.

• Establish QC systems grounded in true comprehension and human judgment, minimizing dependence on LLMs.

• Set and uphold quality standards for training data.

• Create experiments and metrics to evaluate agent outputs.

• Collaborate with data vendors to troubleshoot quality problems and analyze agent failure modes.

• Offer actionable insights to enhance vendor data-generation processes.

• Convert QC insights into supplier-dataset auditing systems, including sampling strategies, validation pipelines, and feedback mechanisms.

• Embed QC insights into infrastructure tools and the data vendor portal to minimize anomalies, inconsistencies, and edge cases.

• Engage in a process that involves two technical interviews and a 2–3 day work trial.

⛳️ Requirements

• Proficiency in Python, Docker, and Linux environments.

• Solid understanding of what constitutes “good data” and methods to measure it.

• A genuine curiosity for diverse domains and the ability to ask insightful questions to grasp them.

• Experience in developing scalable data validation pipelines and automated QA/QC systems from start to finish without a detailed roadmap.

• Background in benchmarks and evaluations, including the ability to reason about realistic tasks, reliable rubrics, usable environments, and effective RL training trajectories.

• Experience in early-stage startups.

• Capacity to work autonomously in fast-paced settings.

• Knowledge of statistics is a plus.

• Strong written and verbal communication skills are advantageous.

• Comfort in designing metrics, experiments, and QA/QC processes is beneficial.

• Experience with existing benchmarks and the creation of tasks in new evaluations is advantageous.

• Ability to excel in unstructured problem environments.

• Willingness to work hours that overlap 70–80% with either the San Francisco or Singapore time zones for remote work.

• Technical aptitude and potential for learning are valued over years of experience.

🏝️ Benefits

• Comprehensive coverage for top-tier medical, dental, and vision plans from Blue Shield of CA (for US employees).

• Complimentary lunch and dinner for in-office employees.

• Company-wide holiday break (from Christmas Eve to New Year’s Day) in addition to PTO and paid holidays.

• Paid time off and holidays.

• Equinox membership (for US employees).

• 401k plan (for US employees).

• Commuter benefits (for US employees).

• Unlimited access to tokens for tools like ChatGPT, Claude Code, Cursor, etc.

• Support for relocation and visa assistance for strong full-time candidates moving to the US or Singapore.