Lead QA Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and implement the evaluation framework for LLM and agentic AI outputs.

• Create assertions, establish behavioral contracts, and manage regression baselines for model performance.

• Construct and update the automated testing suite for backend APIs and frontend interfaces.

• Lead performance and load testing for latency-sensitive inference pathways.

• Generate test artifacts, audit logs, and process documentation that comply with SOC 2 Type II standards.

• Collaborate directly with Forward Deployed Engineering on client-side validation and reproduction of production issues.

⛳️ Requirements

• A minimum of seven years in software QA engineering, including at least two years specifically testing AI or ML systems.

• Proficiency in Python and experience in creating automated suites using pytest, Playwright, or Selenium.

• Practical experience with RAGAS, DeepEval, LangSmith, or similar evaluation tools.

• Capability to trace a failure from the application level to the infrastructure layer.

• Integrated QA gates into CI/CD pipelines and managed the process from start to finish.

• Experience in fintech, banking, or other regulated sectors is highly beneficial.

• Knowledge of document processing pipelines, multi-agent architectures, RAG validation, or observability tools such as Arize or Langfuse.

🏝️ Benefits

• Competitive base salary and significant equity opportunities.

• Remote work (within the US) with occasional travel to client locations and team offsite meetings.