AI Quality and Content Specialist

This is a fully remote position, open to applicants in Canada.

📋 Description

• Conduct audits of AI-generated responses across chat interactions and self-service flows.

• Detect hallucinations, assumed resolutions, incomplete answers, and compliance risks.

• Evaluate AI performance using established QA frameworks.

• Review and validate knowledge content, workflows, and source materials utilized by AI systems.

• Identify outdated, conflicting, incomplete, or low-confidence content that impacts AI responses and customer experience.

• Develop ongoing content governance processes to ensure AI readiness.

• Analyze the reasons behind AI failures, including content gaps, ineffective prompt handling, and edge cases.

• Provide actionable recommendations to the Content Strategist and Head of Support.

• Establish a feedback loop with Product and Engineering teams, representing the voice of the customer.

• Design and maintain dashboards that track resolution accuracy, escalation drivers, and customer sentiment.

• Highlight trends and suggest improvements.

• Collaborate with stakeholders to retrain AI, enhance knowledge content, and modify workflows and guardrails.

• Assist in creating a closed-loop AI optimization system.

• Continuously refine and uphold the quality of AI source content in collaboration with Content, Product, Engineering, and Support teams.

• Develop scalable auditing mechanisms to pinpoint content gaps, outdated documentation, and declining AI confidence or resolution quality.

• Help establish standards for AI-compatible knowledge design.

• Aim to reduce AI error rates, enhance AI CSAT and containment, boost AI trust and adoption, and create an AI governance QA framework.

⛳️ Requirements

• Must be eligible to work without the need for sponsorship.

• Proven experience in auditing AI-generated responses and identifying hallucinations, assumed resolutions, incomplete answers, and compliance risks.

• Proficiency in scoring AI performance against established QA frameworks.

• Experience in auditing and validating knowledge content, workflows, and source materials used by AI systems.

• Ability to identify outdated, conflicting, incomplete, or low-confidence content.

• Skill in establishing content governance processes for help center articles, macros, internal documentation, and workflow logic.

• Strong root cause analysis abilities related to content gaps, poor prompt handling, and edge cases.

• Capability to build and maintain dashboards that monitor resolution accuracy, escalation drivers, and customer sentiment.

• Ability to collaborate with Content, Product, Engineering, and Support stakeholders.

• Experience in creating scalable auditing mechanisms and defining standards for AI-compatible knowledge design.

• Willingness and capability to travel to Denver, Colorado, or other office locations across North America if necessary.

🏝️ Benefits

• Flexibility to work in a manner that suits you within your country of employment – whether in-office, remote, or hybrid.

• Comprehensive wellness benefits, including an annual wellness stipend.

• Three weeks of vacation to start.

• Paid volunteer day of your choosing.

• Two floating holidays to celebrate any significant occasion for you.

• Employee Stock Purchase Program.

• Registered Retirement Savings Plan (RRSP) with a 4% company match.