
Senior Machine Learning Engineer, Evaluation, Reliability
Posted 5 days ago

Posted 5 days ago
This is a fully remote position, open to applicants in United States.
• Create and implement evaluation infrastructure for Bask Health’s AI and machine learning solutions.
• Develop automated testing frameworks for LLM applications and intelligent workflows.
• Establish quantifiable quality benchmarks and evaluation pipelines that assess accuracy, relevance, completeness, consistency, and reliability.
• Construct and maintain representative benchmark datasets and regression testing frameworks.
• Design evaluation strategies for retrieval-augmented generation and knowledge-grounded AI systems.
• Assess retrieval quality, grounding accuracy, and unsupported model outputs.
• Develop tools to detect hallucinations, faulty reasoning, and unreliable responses.
• Create testing methodologies for AI agents, tool integration, and multi-step workflows.
• Implement automated checks for structured outputs, API interactions, and tool execution.
• Develop systems for production monitoring, dashboards, and reporting tools focused on model quality, latency, errors, and operational reliability.
• Investigate failures in production AI and determine the underlying technical causes.
• Collaborate with applied AI engineers to enhance prompts, retrieval pipelines, and model configurations.
• Set up testing protocols for model upgrades and provider transitions.
• Develop benchmarking tools to compare models based on quality, performance, and cost.
• Implement evaluation processes for sensitive healthcare-related AI workflows.
• Work alongside product and clinical stakeholders to define acceptance criteria and escalation protocols.
• Facilitate human reviews and feedback collection for AI outputs that require additional validation.
• Design reproducible experiments and create reusable evaluation libraries, services, and internal tools.
• Integrate AI evaluation checks within development and deployment workflows.
• Enhance observability through structured logging, tracing, and production diagnostics.
• Mentor engineers in AI testing methodologies and reliability best practices.
• Contribute to discussions around AI infrastructure, testing, and monitoring architecture.
• Take ownership of evaluation and reliability enhancements from technical design through deployment and continuous maintenance.
• Over 5 years of professional experience in machine learning engineering, software engineering, or a closely related technical field.
• Proven experience in developing and operating machine learning or AI systems in a production environment.
• Proficient programming skills in Python and experience in building production software services.
• Experience in designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.
• In-depth understanding of machine learning evaluation methodologies and performance metrics.
• Experience in developing reproducible experiments and benchmark datasets.
• Familiarity with large language models and contemporary LLM application architectures.
• Understanding of prompt engineering, model inference, and AI application integration patterns.
• Experience in evaluating model outputs for quality, accuracy, and reliability.
• Familiarity with retrieval-augmented generation and retrieval evaluation techniques.
• Experience with structured datasets, data pipelines, and analytical workflows.
• Strong understanding of software testing, regression testing, and continuous integration practices.
• Background in building or maintaining backend services and APIs.
• Familiarity with cloud infrastructure and production deployment settings.
• Experience with monitoring, logging, tracing, and troubleshooting in production.
• Understanding of model latency, throughput, and operational performance metrics.
• Ability to investigate complex system failures and identify root causes.
• Experience in designing metrics and reporting systems for technical performance.
• Familiarity with version control, code review, and collaborative software development practices.
• Understanding of data quality and its implications for model evaluation.
• Ability to assess technical trade-offs between quality, performance, reliability, and cost.
• Strong analytical skills and meticulous attention to detail.
• Experience collaborating with engineers, product managers, and technical stakeholders.
• Capacity to independently manage complex engineering projects.
• Excellent communication skills and the ability to convey evaluation results effectively.
• Preferred qualifications include experience with LLM evaluation frameworks, LangSmith, DeepEval, Ragas, LLM-as-a-judge methods, hallucination detection, agent evaluation, tool-calling evaluation, retrieval metrics, vector databases, embeddings, semantic search, MLflow, Weights & Biases, observability infrastructure, model drift detection, CI/CD evaluation checks, distributed processing, AWS, AI developer tools, healthcare applications, HIPAA, human-in-the-loop evaluation, Cursor, and Claude Code.
• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Flexible working hours and remote work opportunities.
• Professional development and continuous learning programs.
• Collaborative and inclusive work environment.
Memory Bridge
Entrada
Find Your Talent (FYT)
Inmetrics
Get handpicked remote jobs straight to your inbox weekly.