Senior Manager – Data Quality and Evaluation

This is a fully remote position, open to applicants in Ireland.

📋 Description

• Develop and oversee quality frameworks for AI data and evaluation initiatives.

• Convert client requirements into quality benchmarks, rubrics, acceptance criteria, review processes, and KPIs.

• Create scalable quality workflows for programs transitioning from pilot to production.

• Recognize quality risks and address issues in collaboration with delivery teams.

• Establish repeatable processes for calibration, QA sampling, adjudication, reviewer performance monitoring, and customer reporting.

• Facilitate quality operations across multilingual evaluations, speech/audio QA, transcription, annotation, human preference evaluation, expert reviews, model response evaluations, coding assessments, tool-use evaluations, and agent workflow assessments.

• Develop and enhance rubrics, task instructions, reviewer guides, calibration exercises, golden datasets, and quality reporting templates.

• Lead calibration sessions with reviewers, annotators, quality specialists, delivery teams, and customer stakeholders.

• Set quality thresholds, error taxonomies, escalation protocols, and corrective action plans.

• Track reviewer agreement, disagreement trends, error rates, contributor performance, and root causes of quality discrepancies.

• Convert QA findings into enhancements for instructions, training, tools, staffing, and delivery workflows.

• Serve as the quality lead for strategic customer programs as necessary.

• Assist in customer-facing quality presentations, pilot retrospectives, business reviews, escalations, and scale-up discussions.

• Deliver data-driven reports on quality performance, risks, corrective actions, and future steps.

• Collaborate with Program Management, Supply Chain, Solutions, Sales, and Operations to ensure quality success.

• Define reviewer profiles, evaluator requirements, language needs, domain expertise, onboarding requirements, and performance expectations.

• Assess when programs necessitate expert reviewers, QA leads, language leads, technical reviewers, or specialized evaluation talent.

• Create reusable quality assets, standardized methodologies, scorecards, and sample evaluation frameworks.

• Enhance transparency into quality performance across programs, reviewers, contributors, and workflows.

• Oversee, mentor, and support Quality Managers, Quality Leads, Quality Specialists, reviewers, and QA contributors.

• Identify hiring, training, and coverage requirements.

• Foster a culture of quality ownership, accountability, and continuous improvement.

• Anticipate and communicate team needs effectively.

• Educate and assist team members, advocate for skill enhancement, and promote career advancement.

• Provide assistance during peak workload periods and arrange coverage for absences.

⛳️ Requirements

• Over 5 years of experience in quality operations, data operations, AI data services, localization quality, annotation quality, evaluation operations, trust and safety quality, or a related field.

• Experience managing quality programs for intricate customer accounts or high-volume operational delivery.

• Comprehensive understanding of QA methodologies, calibration, sampling, adjudication, error analysis, and performance reporting.

• Proven experience working cross-functionally with delivery, operations, supply chain, sales, and customer-facing teams.

• Strong analytical capabilities and the ability to transform quality data into actionable operational improvements.

• Exceptional written and verbal communication skills, including the ability to communicate quality concerns clearly to clients and senior stakeholders.

• Ability to thrive in dynamic, uncertain environments where processes are still being established.

• Strong leadership skills with experience mentoring quality specialists, reviewers, annotators, or operational teams.

• Preferred: experience with AI data, RLHF, model evaluation, LLM evaluation, speech/audio evaluation, transcription, coding evaluation, multilingual evaluation, or expert review programs.

• Preferred: experience designing rubrics, annotation guidelines, evaluation instructions, reviewer training, calibration workflows, or quality scorecards.

• Preferred: experience supporting AI labs, enterprise AI teams, research teams, or technical clients.

• Preferred: familiarity with human-in-the-loop data workflows, annotation platforms, QA tools, dashboards, and data labeling operations.

• Preferred: experience collaborating with expert contributors, linguists, annotators, domain specialists, technical reviewers, or distributed talent networks.

• Preferred: knowledge of multilingual evaluation, speech/audio QA, cultural appropriateness, or language-specific quality risks.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and career advancement.

• Flexible work arrangements and a supportive team environment.

• Access to cutting-edge tools and technologies.