Senior AI Systems Quality Engineer

This is a fully remote position, open to applicants in Massachusetts.

📋 Description

• Develop and deploy production-quality automated validation frameworks, test harnesses, and evaluation pipelines throughout the AI lifecycle.

• Design and enhance an AI testing platform that integrates with Databricks and MLflow.

• Create extensive scenario-based test suites to assess agentic workflows, edge cases, long-tail scenarios, and potential failure modes.

• Validate orchestration behavior, including tool utilization, memory management, and decision-making logic.

• Conduct stress tests on non-deterministic system behaviors prior to production deployment.

• Establish system contracts, guardrails, and safe-degradation patterns.

• Define quantifiable LLM quality signals such as grounding, hallucinations, relevance, latency, and cost.

• Integrate automated quality gates into CI/CD pipelines.

• Ensure that AI validation processes run automatically with model, prompt, and code modifications.

• Develop reusable libraries and components to maintain consistent AI quality practices.

• Oversee aspects of AI release readiness and determine go/no-go criteria.

• Collaborate with AI, platform, security, and delivery teams to establish quality standards, trade-offs, and confidence levels.

⛳️ Requirements

• Over 7 years of software engineering experience, with a focus on backend or platform systems.

• Demonstrated experience in designing and implementing AI testing automation within production settings.

• Proven ability to create custom validation, evaluation, or testing frameworks for complex, distributed systems.

• Strong command of Python and/or TypeScript in contemporary AI engineering frameworks.

• Practical experience with AI-powered systems, including LLM-based or agentic workflows and non-deterministic behaviors.

• Experience in designing or contributing to large-scale AI testing, including regression frameworks, long-tail evaluations, and extensive test coverage.

• In-depth knowledge of CI/CD integration, including the incorporation of automated tests and quality gates into deployment pipelines.

• Solid comprehension of AWS cloud-native architectures.

• Proven track record of engineering with quality, reliability, governance, and safety as fundamental design principles.

• Familiarity with security, privacy, and operational risks in regulated or mission-critical settings, including failure modes and recovery processes.

• Experience with AI testing methodologies, encompassing evaluation of non-deterministic outputs, drift detection, bias/fairness testing, and robust regression strategies.

• Demonstrated ability to set measurable trust thresholds for AI systems, such as query accuracy, hallucination limits, explainability, and PHI-safe behavior as enforceable release criteria.

• Experience collaborating with domain experts to define correctness and real-world validation scenarios.

• Familiarity with Databricks-native environments and Medallion architecture is preferred but not mandatory.

• Preferred experience with MLflow, observability tools, LLM evaluation techniques, guardrails, policy enforcement frameworks, AI performance/latency/cost regression evaluation, documentation, formal AI/ML training or certification, and AI-generated test scenarios.

🏝️ Benefits

• Unlimited paid time off – recharge as needed.

• Work from anywhere – flexibility to accommodate your lifestyle.

• Comprehensive health coverage – multiple plan options available.

• Equity for every employee – participate in our success.

• Growth-oriented environment – your development is valued here.

• Home office setup allowance – one-time support to help you get started.

• Monthly cell phone allowance – stay connected effortlessly.

• Eligibility for performance bonuses.

• Equity grants.