Project Lead, AI Model Training

This is a fully remote position, open to applicants in United States, +2 more locations.

📋 Description

• Take ownership of client engagements from initial scoping to delivery on NewtonX’s data operations platform.

• Convert client training or evaluation requirements into comprehensive statements of work, task designs, quality benchmarks, annotator profiles, and strategies for agentic-project environments/tools.

• Set up multi-layer quality pipelines, task forms, grading rubrics, annotator enablement, and routing processes.

• Collaborate with the ML Lead to draft initial guidelines and gold sets, rigorously testing them prior to production.

• Monitor project quality, throughput, and costs against established targets; identify deviations and take corrective actions before they impact the client.

• Oversee annotator assignments, access layers, qualifications, and performance in coordination with the annotator delivery team.

• Facilitate project channels and calibration sessions, manage exceptions and escalations, and modify configurations as needed.

• Act as the main client liaison for scoping discussions, updates, quality escalations, and delivery processes.

• Align expectations with project scope and identify any acceptance issues.

• Highlight opportunities for account expansion and new use cases to the commercial team.

• Ensure timely assembly and delivery of batches, managing the client handoff process.

• Conduct client acceptance testing, obtain sign-offs, and initiate commercial milestones.

• Carry out post-mortem analyses and integrate insights into guidelines, tools, and future project scopes.

• Develop repeatable delivery playbooks, templates, and quality standards.

• Relay delivery needs and platform feedback to product and engineering teams.

⛳️ Requirements

• A minimum of 3 years of experience in data labeling, RLHF/SFT, or AI evaluation delivery.

• Proven track record of delivering actual annotation or evaluation projects within a data labeling firm or an AI lab.

• Proficient in owning quality metrics, troubleshooting stalled queues, and revising production rubrics.

• Capable of conducting scoping conversations with technical buyers and translating operational processes into terminology relevant to clients.

• Strong data analysis abilities using SQL or Python are highly desirable.

• Experience in creating processes, playbooks, templates, and quality standards.

• Competent in managing multiple concurrent projects with varying clients, quality standards, and annotator groups.

• Demonstrates adaptability and resilience in dynamic project settings.

• Maintains discretion in handling annotator personal data and client confidential information in accordance with strict data security and confidentiality protocols.

• Valid work authorization for the US or Canada is required.

• Visa sponsorship is not available for this position.

🏝️ Benefits

• Participation in our company stock buying program.

• Comprehensive medical, dental, and vision insurance.

• 401k matching with immediate vesting.

• Health savings and flexible spending accounts.

• Pre-tax commuter benefits.

• Paid time off, including vacation, holidays, sick leave, and parental leave.

• Accelerated career progression towards operational and commercial leadership roles.

• A diverse, collaborative, and positive workplace culture.

• Engaging activities such as happy hours, team projects, and retreats.