Senior Machine Learning Engineer, Evaluation, Reliability

Posted 5 days ago

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and implement evaluation infrastructure for Bask Health’s AI and machine learning solutions.

• Develop automated testing frameworks for LLM applications and intelligent workflows.

• Establish quantifiable quality benchmarks and evaluation pipelines that assess accuracy, relevance, completeness, consistency, and reliability.

• Construct and maintain representative benchmark datasets and regression testing frameworks.

• Design evaluation strategies for retrieval-augmented generation and knowledge-grounded AI systems.

• Assess retrieval quality, grounding accuracy, and unsupported model outputs.

• Develop tools to detect hallucinations, faulty reasoning, and unreliable responses.

• Create testing methodologies for AI agents, tool integration, and multi-step workflows.

• Implement automated checks for structured outputs, API interactions, and tool execution.

• Develop systems for production monitoring, dashboards, and reporting tools focused on model quality, latency, errors, and operational reliability.

• Investigate failures in production AI and determine the underlying technical causes.

• Collaborate with applied AI engineers to enhance prompts, retrieval pipelines, and model configurations.

• Set up testing protocols for model upgrades and provider transitions.

• Develop benchmarking tools to compare models based on quality, performance, and cost.

• Implement evaluation processes for sensitive healthcare-related AI workflows.

• Work alongside product and clinical stakeholders to define acceptance criteria and escalation protocols.

• Facilitate human reviews and feedback collection for AI outputs that require additional validation.

• Design reproducible experiments and create reusable evaluation libraries, services, and internal tools.

• Integrate AI evaluation checks within development and deployment workflows.

• Enhance observability through structured logging, tracing, and production diagnostics.

• Mentor engineers in AI testing methodologies and reliability best practices.

• Contribute to discussions around AI infrastructure, testing, and monitoring architecture.

• Take ownership of evaluation and reliability enhancements from technical design through deployment and continuous maintenance.


⛳️ Requirements

• Over 5 years of professional experience in machine learning engineering, software engineering, or a closely related technical field.

• Proven experience in developing and operating machine learning or AI systems in a production environment.

• Proficient programming skills in Python and experience in building production software services.

• Experience in designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.

• In-depth understanding of machine learning evaluation methodologies and performance metrics.

• Experience in developing reproducible experiments and benchmark datasets.

• Familiarity with large language models and contemporary LLM application architectures.

• Understanding of prompt engineering, model inference, and AI application integration patterns.

• Experience in evaluating model outputs for quality, accuracy, and reliability.

• Familiarity with retrieval-augmented generation and retrieval evaluation techniques.

• Experience with structured datasets, data pipelines, and analytical workflows.

• Strong understanding of software testing, regression testing, and continuous integration practices.

• Background in building or maintaining backend services and APIs.

• Familiarity with cloud infrastructure and production deployment settings.

• Experience with monitoring, logging, tracing, and troubleshooting in production.

• Understanding of model latency, throughput, and operational performance metrics.

• Ability to investigate complex system failures and identify root causes.

• Experience in designing metrics and reporting systems for technical performance.

• Familiarity with version control, code review, and collaborative software development practices.

• Understanding of data quality and its implications for model evaluation.

• Ability to assess technical trade-offs between quality, performance, reliability, and cost.

• Strong analytical skills and meticulous attention to detail.

• Experience collaborating with engineers, product managers, and technical stakeholders.

• Capacity to independently manage complex engineering projects.

• Excellent communication skills and the ability to convey evaluation results effectively.

• Preferred qualifications include experience with LLM evaluation frameworks, LangSmith, DeepEval, Ragas, LLM-as-a-judge methods, hallucination detection, agent evaluation, tool-calling evaluation, retrieval metrics, vector databases, embeddings, semantic search, MLflow, Weights & Biases, observability infrastructure, model drift detection, CI/CD evaluation checks, distributed processing, AWS, AI developer tools, healthcare applications, HIPAA, human-in-the-loop evaluation, Cursor, and Claude Code.


🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work opportunities.

• Professional development and continuous learning programs.

• Collaborative and inclusive work environment.

People also viewed

Memory Bridge10 hours ago

Senior Applied ML Engineer

US flagUnited States OnlyFull-timeMachine Learning Engineer
ApplyView job
Entrada12 hours ago

Senior Machine Learning Engineer, Databricks

PL flagPoland, +2 more countriesFull-timeMachine Learning Engineer
ApplyView job
Find Your Talent (FYT)14 hours ago

Senior Machine Learning Engineer

US flagUnited States OnlyFull-timeMachine Learning Engineer
ApplyView job
Inmetrics14 hours ago

Machine Learning Engineer

BR flagBrazil OnlyFull-timeMachine Learning Engineer
ApplyView job
SandboxAQ15 hours ago

Staff Machine Learning Engineer – Medical Devices

CA flagCanada OnlyFull-timeMachine Learning EngineerC$146k – C$258k/year
ApplyView job
StackAdapt15 hours ago

Staff Applied Machine Learning Scientist – Identity

US flagUnited States, +1 more countryFull-timeMachine Learning Engineer$182.4k – $250.8k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers