Strategic Projects Lead

📋 Description

• Create comprehensive data collection and evaluation pipelines for RLVR, RLHF, SFT, red-teaming, and model assessment workflows.

• Develop, test, and refine AI agents that automate tasks within the pipeline — including quality gate reviews, expert matching, output flagging, and detection of anomalies in throughput.

• Establish data quality benchmarks across annotation, evaluation, and expert output reviews.

• Collaborate directly with AI researchers, Technical Program Managers (TPMs), and Product Managers (PMs) at client organizations.

• Remain informed about advancements in LLM post-training, evaluation methodologies, and data tooling.

⛳️ Requirements

• Expertise in Python and SQL for data manipulation, monitoring pipelines, and conducting quality analyses.

• Familiarity with LLM internals: RLHF/SFT training loops, the impact of prompt structure on output distribution, and RL environment setup characteristics (tool use) for agentic data collection and evaluation projects.

• Practical experience with at least one agentic or LLM workflow framework (such as LangChain, DSPy, AutoGen, or direct tool use via API, or similar).

• Proven track record of owning a data or machine learning pipeline from initial scoping to final delivery — including the design of quality measures, not solely throughput monitoring.

• Excellent written communication skills: you will draft technical guidelines and rubrics for distributed expert workers to follow accurately, and you will update senior researchers on pipeline performance.

• Ability to navigate ambiguity in a dynamic environment where model requirements change and client priorities shift.

🏝️ Benefits

• Equity offerings.