Senior Manager, Data Quality – Evaluation

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop high-quality systems for AI data initiatives.

• Design and oversee quality frameworks for AI data and evaluation initiatives.

• Convert customer needs into well-defined quality standards, rubrics, acceptance criteria, review processes, and KPIs.

• Establish quality workflows that are practical, scalable, and trusted by clients as programs transition from pilot to production.

• Proactively identify quality risks and collaborate with delivery teams to resolve issues before they affect timelines, customer confidence, or program results.

• Develop repeatable quality processes across calibration, QA sampling, adjudication, reviewer performance tracking, and client reporting.

• Lead evaluation, calibration, and QA initiatives.

• Support quality operations in multilingual evaluations, speech/audio QA, transcription, data annotation, human preference evaluation, expert review, model response evaluation, coding evaluation, tool-use evaluation, and agent workflow evaluation.

• Create and refine rubrics, task instructions, reviewer guides, calibration exercises, golden datasets, and quality reporting templates.

• Facilitate calibration sessions with reviewers, annotators, quality specialists, delivery teams, and client stakeholders.

• Establish quality thresholds, error taxonomies, escalation procedures, and corrective action plans.

• Track reviewer agreement, disagreement trends, error rates, contributor performance, and the root causes of quality variances.

• Transform QA findings into actionable enhancements for instructions, training, tooling, staffing, and delivery workflows.

• Collaborate with clients and internal teams.

• Serve as a quality lead for strategic customer programs when necessary.

• Assist with customer-facing quality readouts, pilot retrospectives, business reviews, escalations, and discussions on scaling.

• Provide clear, data-driven reports that outline quality performance, risks, corrective actions, and upcoming steps.

• Collaborate with Program Management, Supply Chain, Solutions, Sales, and Operations to ensure programs are primed for quality success from the outset.

• Work with Supply Chain to define reviewer profiles, evaluator needs, language requirements, domain expertise, onboarding necessities, and performance expectations.

• Determine when programs necessitate expert reviewers, QA leads, language leads, technical reviewers, or specialized evaluation talent.

• Build and enhance the quality function.

• Develop reusable quality assets such as calibration packs, QA reports, rubric libraries, error taxonomies, scorecards, and sample evaluation frameworks.

• Identify repeatable trends across programs and convert them into standardized approaches that facilitate business scalability.

• Enhance visibility into quality performance across programs, reviewers, contributors, and workflows.

• Manage, mentor, and support Quality Managers, Quality Leads, Quality Specialists, reviewers, or QA contributors assigned to Data Services programs.

• Mentor team members on quality judgment, customer communication, escalation management, reporting, and root-cause analysis.

• Recognize hiring, training, and coverage needs as the Data Services business expands.

• Foster a culture of quality ownership, accountability, and continual improvement.

• Anticipate and communicate the needs identified for the team under your supervision.

• Train and support team members, advocate for skill enhancement, and promote career advancement.

• Be responsible for assisting team members during peak workload periods, helping to mitigate risks to Client deliveries due to time constraints (this includes arranging cover for sickness and absence).

⛳️ Requirements

• Over 5 years of experience in quality operations, data operations, AI data services, localization quality, annotation quality, evaluation operations, trust and safety quality, or a related discipline.

• Experience in managing quality programs for intricate customer accounts or high-volume operational delivery.

• Strong grasp of QA methodologies, calibration, sampling, adjudication, error analysis, and performance reporting.

• Experience collaborating cross-functionally with delivery, operations, supply chain, sales, and customer-facing teams.

• Strong analytical skills with the capacity to convert quality data into clear operational enhancements.

• Excellent written and verbal communication skills, including the capacity to communicate quality issues effectively to customers and senior stakeholders.

• Comfortable working in fast-paced, ambiguous environments where processes are still evolving.

• Strong leadership abilities with experience in coaching quality specialists, reviewers, annotators, or operational teams.

🏝️ Benefits

• Opportunities for professional development.