AI Evaluation Engineer

This is a fully remote position, open to applicants in Florida.

📋 Description

• Design, construct, and manage evaluation systems for Generative AI solutions.

• Create standardized, automated evaluation frameworks for both large language models (LLMs) and traditional machine learning models.

• Implement observability solutions to monitor model drift, hallucination risks, latency, throughput, and token usage.

• Define methodologies and metrics for evaluating LLMs, retrieval-augmented generation (RAG), and agent-based applications.

• Develop reusable evaluation frameworks, templates, and libraries.

• Convert business requirements into quantifiable quality criteria and acceptance benchmarks.

• Create automated evaluation pipelines that integrate into CI/CD with regression testing and quality gates.

• Execute both offline and online evaluations, including benchmarks, golden datasets, production monitoring, sampling, drift detection, and quality alerts.

• Facilitate A/B and shadow testing for model, prompt, and architectural modifications.

• Design and sustain LLM-as-a-judge evaluators, focusing on rubric design, bias mitigation, and human-label calibration.

• Establish human annotation workflows and labeling protocols.

• Develop golden datasets, adversarial and edge-case suites, and synthetic test data generation methods.

• Implement data handling practices, including de-identification, synthetic data usage, and access controls.

• Conduct red-teaming, hallucination, toxicity, personally identifiable information (PII) leakage, prompt-injection, and jailbreak testing.

• Generate evaluation evidence and produce reports for security, privacy, compliance, and model risk assessments.

• Contribute to the establishment of responsible AI standards and release-readiness criteria.

• Collaborate closely with product teams to onboard them to evaluation practices and guide them toward ownership.

• Create dashboards and reports that track quality and cost/latency trends.

• Assess and recommend evaluation tools and platforms; document standards and best practices.

• Perform additional duties as assigned.

• Travel as necessary.

⛳️ Requirements

• Master's degree in computer science or a related quantitative field from an accredited institution is preferred.

• 4+ years of experience in software, machine learning, or data engineering.

• At least 1-2 years of hands-on experience with LLM-based applications.

• Professional experience with evaluation and observability frameworks such as LangSmith, Arize Phoenix, or WhyLabs.

• Strong comprehension of statistical validation and model evaluation metrics.

• Proficient in Python and possess foundational software engineering skills, including testing, version control, CI/CD, and code review.

• Practical experience in evaluating LLM/RAG systems, including retrieval metrics like recall, precision, MRR/nDCG, and generation metrics.

• Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.

• Working knowledge of statistics and experimental design principles, including sampling, confidence intervals, significance testing, and inter-rater agreement.

• Familiarity with at least one evaluation or observability tool, such as Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or similar.

• Experience with cloud services like Azure, AWS, or GCP and their managed LLM offerings.

• Strong communication abilities and capacity to collaborate with engineers, product owners, subject matter experts (SMEs), and risk stakeholders.

• Excellent verbal and written communication skills, including presentation capabilities.

• Ability to work effectively in a team environment.

• Capability to meet or surpass performance competencies.

• Preferred qualifications include experience in evaluating agentic systems, a background in ML evaluation/search/relevance/NLP/QA automation, LLMOps/monitoring/observability experience, and familiarity with AI risk and governance frameworks, data annotation workflows, and enterprise controls.

🏝️ Benefits

• Work-life balance.

• Reasonable accommodations when applicable and appropriate.

• Equal Opportunity Employer.

• Drug-Free Workplace.