Inference QA Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Take ownership of and enhance tiered evaluation frameworks for data retrieval, statistical analysis, open-ended inference, and root-cause analysis.

• Create test sets that include ground-truth rubrics, pass criteria, known failure modes, and source tables.

• Execute large-scale prompts against live model endpoints on a scheduled basis as well as on demand.

• Maintain the test harness and convert raw model executions into deployment verdicts.

• Develop observability and monitoring systems for multi-step agentic configurations.

• Proactively identify silent empty responses, false refusals, tool-routing misses, latency problems, non-termination, and instability.

• Collaborate with subject-matter experts to extract verified ground truths from deployment feedback.

• Transform confirmed defects into permanent regression safeguards.

• Design tier-appropriate tests after implementing fixes, execute them against deployments, and report pass rates compared to established thresholds.

• Manage the inference-quality gate for production deployments.

⛳️ Requirements

• Minimum of 5 years of experience in software, ML, data, or QA engineering, with responsibility for a quality-critical system.

• Proficient in Python programming.

• Comfortable working with FastAPI, Postgres, and Docker.

• Capable of reading logs across services and tracing requests through distributed pipelines.

• Knowledgeable in LLM prompting, tool/function calling, agentic loops, RAG, hallucinations, refusals, silent truncation, and non-determinism.

• Experience in designing LLM evaluations, deterministic checks, ground-truth scoring, and statistical consistency measures.

• Proficient in SQL, including assessing generated queries and table selections.

• Experience with monitoring and observability, including dashboards, alerts, and trace inspection.

• Strong capability in managing uncertainty and calibrated ranges.

• Bonus: experience in evaluating or red-teaming agentic or multi-tool LLM systems.

• Bonus: familiarity with MLflow or similar tools for trace and experiment management.

• Bonus: experience with technical end users and converting feedback into reproducible tests.

• Bonus: background in time-series, forecasting, industrial, or operational data.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and remote work options.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive work environment.