AI Implementation Quality Analyst

This is a fully remote position, open to applicants in United States.

📋 Description

• Collaborate closely with clients during implementation to comprehend use cases, success metrics, and risk tolerance.

• Convert client specifications into evaluation frameworks, prompt strategies, and testing coverage.

• Serve as the quality intermediary between the client, product team, and engineering to ensure consistency both pre- and post-launch.

• Assess and score AI responses using established rubrics.

• Validate against sources/ground truth; identify hallucinations, omissions, and other potential risks.

• Document observations and calibrate with colleagues to maintain consistent scoring practices.

• Develop and sustain prompt banks and golden sets (expected outcomes).

• Broaden coverage for edge cases, high-risk situations, and authentic user language.

• Monitor regression/drift and provide data for dashboards and quality reports.

• Prioritize internal and client feedback; distill themes across various deployments.

• Recognize systemic risks and escalate significant findings with clear, client-relevant context.

• Collaborate with product and engineering teams to validate and confirm fixes, ensuring alignment with client expectations.

⛳️ Requirements

• Strong analytical judgment and a keen attention to detail.

• Exceptional written communication skills, including the ability to articulate reasoning clearly.

• Experience in reviewing, auditing, or assessing structured outputs such as content, decisions, or recommendations.

• Comfort with applying detailed guidelines and rubrics consistently at scale.

• Familiarity with large language models and common failure modes like hallucinations, overgeneralization, or unsafe responses.

• Proficiency in using spreadsheets, evaluation tools, or annotation platforms.

• Experience with AI evaluation, data annotation, quality assurance, trust and safety, or policy review.

• Exposure to human-in-the-loop workflows or benchmarking procedures.

• Domain expertise in a regulated or high-risk sector such as government, education, healthcare, or legal services.

• Experience in contributing to test suites, evaluation dashboards, or quality reporting.

• Ability to work collaboratively across functions with product and engineering teams.

• Experience in client-facing roles such as implementation, consulting, or solution delivery in a SaaS or regulated environment.

• Capacity to translate between business/user needs and technical system behaviors.

🏝️ Benefits

• Flexible Time Off – Take the time you need to rest, recharge, and live your life.

• Company-Wide Wellbeing Days – Paid days off to unplug and focus on your mental health.

• Work From Home Reimbursement – Support a productive home office environment.

• Multiple Health Plan Options – Including a 100% employer-paid plan.

• Employer HSA Contributions – When enrolled in a High-Deductible Health Plan.

• Fitness Reimbursement Program – Stay active, your way.

• On-Demand Mental Health Support – Access to Headspace and other wellness tools.

• Paid Parental Leave – For both birthing and non-birthing parents.

• Traditional & Roth 401(k) – With a generous company match.

• Life & AD&D Insurance – 100% employer-paid coverage for peace of mind.

• Online Learning Platforms – Fuel your professional development.

• Competitive Salary & Bonuses – Your contributions are valued and rewarded.