Senior QA Engineer – AI

This is a fully remote position, open to applicants in India.

📋 Description

• Develop and sustain high-quality datasets for AI products, derived from production traces or created and validated against a reliable source of truth.

• Create multi-layered scoring systems for AI outputs utilizing deterministic rules and model-graded assessments.

• Oversee evaluation processes from start to finish, which includes scheduling, execution, baseline comparison, and troubleshooting failures.

• Maintain and manage the release gate for any prompt or model modifications.

• Detect failure modes such as hallucination, improper tool selection, loss of conversational context, and exposure of personally identifiable information (PII).

• Analyze production traces to differentiate between retrieval failures, generation issues, and tool-related errors.

• Transform insights into actionable engineering documentation and ensure lasting regression coverage.

• Establish and communicate quality metrics for AI interfaces and promote continuous improvement.

⛳️ Requirements

• Minimum of 1 year of experience managing quality for AI products in a lead or primary role.

• Proven experience with prompts, tool and function schemas, and understanding model behavior.

• Practical experience in constructing or managing an evaluation harness for an AI system, employing frameworks like DeepEval, RAGAS, or Promptfoo, or an in-house alternative.

• 4-6 years of experience in QA / SDET roles.

• Proficient in Python programming.

• Capability to analyze traces and tool calls with observability tools such as Opik, Langfuse, LangSmith, or similar solutions.

• Strong experience in API testing.

• Adequate engineering skills to develop tooling and automation solutions.

• Excellent written communication abilities.

• Familiarity with trading or exchange platforms, including futures and options, crypto derivatives, margin, or settlement processes is a plus.

🏝️ Benefits

• Competitive salary package.

• Ample opportunities for professional growth.

• Exposure to leadership roles.

• Collaborate directly with founders and senior leadership on impactful strategic projects.

• Chance to influence the regulatory environment for digital assets and Web3 in India.

• Join a mission-driven, rapidly growing organization that is at the cutting edge of financial innovation.