QA Lead, AI Agents

This is a fully remote position, open to applicants in India.

📋 Description

• Lead the Quality Assurance team by establishing audit standards and scorecards, reviewing team members' work, coaching analysts, and determining escalation paths.

• Collaborate with Agent Engineers to provide transcript and audit evidence, verify solutions, and ensure they are effective in production.

• Review call and chat transcripts in Reflect and assess AI-agent performance against established quality and compliance criteria.

• Identify regressions, compliance issues, mishandled workflows, and edge cases, documenting problems using transcript or metric evidence.

• Convert findings into actionable Linear cards, including reproduction steps, affected conversations, and the impact on clients.

• Participate in client QA and audit calls, presenting findings and addressing inquiries regarding agent performance.

• Align internal QA findings with client audit team results to create a cohesive and defensible quality assessment.

• Provide insights, recurring issues, and trends to the Technical Program/Product Manager for release communications, weekly sync-ups, and monthly reviews.

⛳️ Requirements

• Minimum of 2 years of experience in QA, operations, quality analysis, delivery support, or a related client-facing position, preferably within a B2B SaaS, financial services, or consulting environment.

• Proven experience in leading or mentoring a QA or operations team, which includes establishing standards, developing processes, reviewing work, and ensuring quality maintenance.

• Ability to utilize data and transcripts as evidence to independently assess and support findings.

• Proficient in written and spoken English.

• Consistent daily availability that aligns with US business hours (ET/CT), including a late-afternoon-to-evening IST schedule.

• Comfortable participating in discussions with client operations or compliance teams.

• Strong attention to detail, organization, precision, and follow-through.

• Experience in debt collections, lending, or another regulated financial services domain, or familiarity with FDCPA/compliance workflows is a plus.

• Practical experience with QA tools, call-quality scoring, or observability/audit dashboards is advantageous.

• Background in developing or managing testing and auditing frameworks, such as test plans, regression suites, QA scorecards, or compliance audit programs is beneficial.

• Interest in AI and conversational systems, particularly their behavior in production settings, is a plus.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional development and career advancement.

• Collaborative and innovative work environment.

• Flexible working hours and remote work options.