Machine Learning Engineer, Evaluation, Reliability

Posted 4 days ago

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and uphold evaluation systems for AI-driven applications at Bask Health.

• Create automated testing frameworks for extensive language models and AI application workflows.

• Generate and sustain representative datasets for assessing model performance.

• Establish benchmarks that evaluate response accuracy, relevance, completeness, and consistency.

• Design evaluation pipelines for prompts, retrieval systems, and model configurations.

• Assess AI assistants and agents through realistic user scenarios and workflow requirements.

• Construct automated regression tests to identify performance changes across releases.

• Analyze hallucination rates, unsupported claims, and other quality failures of models.

• Assess retrieval-augmented generation systems for document relevance and answer grounding.

• Formulate tests for structured model outputs and schema compliance.

• Evaluate tool-calling accuracy and the selection of agent actions.

• Examine AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.

• Conduct evaluation checks for sensitive patient and operational information.

• Develop dashboards and reports that convey AI quality metrics to engineering teams.

• Analyze evaluation outcomes and recognize recurring failure patterns.

• Investigate production incidents related to inaccurate or unreliable AI outputs.

• Collaborate with applied AI engineers to enhance prompts, retrieval logic, and application functionality.

• Assist in model comparisons and controlled experiments.

• Assess trade-offs among model quality, latency, reliability, and inference costs.

• Create automated quality checks for CI/CD and AI feature release workflows.

• Implement monitoring and alerting systems for production AI performance.

• Develop tools for reviewing, labeling, and analyzing model responses.

• Support the maintenance of evaluation datasets as products and user behaviors evolve.

• Document evaluation methodologies, test cases, and known limitations of models.

• Collaborate with product teams to establish measurable acceptance criteria for AI features.

• Work alongside senior engineers on evaluation standards and improvements in reliability.

• Engage in code reviews, technical discussions, and engineering planning.

• Take ownership of designated evaluation and reliability projects from implementation to deployment and continuous enhancement.


⛳️ Requirements

• Over 3 years of professional experience in machine learning engineering, software engineering, applied AI, or a related technical discipline.

• Proven experience in building and maintaining production software or machine learning applications.

• Proficient programming skills in Python, TypeScript, or similar languages.

• Knowledge of large language models and contemporary generative AI applications.

• Experience in testing or evaluating machine learning models or AI-driven software.

• Familiarity with machine learning evaluation metrics and experimental methodologies.

• Experience in developing automated tests and validation pipelines.

• Understanding of prompt engineering and analysis of model outputs.

• Ability to analyze both structured and unstructured datasets.

• Proven experience in scripting or creating services for data processing and evaluation.

• Knowledge of model accuracy, precision, recall, and other applicable performance measures.

• Familiarity with API integrations and backend application development.

• Experience with relational databases or analytical data systems.

• Ability to recognize patterns in model failures and convert findings into engineering enhancements.

• Understanding of regression testing and software quality assurance practices.

• Familiarity with logging, monitoring, and troubleshooting in production environments.

• Experience utilizing Git and collaborative software development methodologies.

• Awareness of the limitations and failure modes associated with large language models.

• Ability to design representative test cases and evaluation scenarios.

• Familiarity with cloud-based development and deployment platforms.

• Proven ability to write maintainable, well-tested production code.

• Strong analytical skills and attention to detail.

• Ability to communicate technical evaluation results effectively.

• Experience working collaboratively with engineers, product managers, and cross-functional teams.

• Capability to independently manage assigned technical projects.


🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health insurance plans.

• Flexible work hours and remote work options.

• Opportunities for professional development and continuing education.

• Engaging and inclusive work environment.

People also viewed

Memory Bridge10 hours ago

Senior Applied ML Engineer

US flagUnited States OnlyFull-timeMachine Learning Engineer
ApplyView job
Entrada11 hours ago

Senior Machine Learning Engineer, Databricks

PL flagPoland, +2 more countriesFull-timeMachine Learning Engineer
ApplyView job
Find Your Talent (FYT)13 hours ago

Senior Machine Learning Engineer

US flagUnited States OnlyFull-timeMachine Learning Engineer
ApplyView job
Inmetrics13 hours ago

Machine Learning Engineer

BR flagBrazil OnlyFull-timeMachine Learning Engineer
ApplyView job
SandboxAQ15 hours ago

Staff Machine Learning Engineer – Medical Devices

CA flagCanada OnlyFull-timeMachine Learning EngineerC$146k – C$258k/year
ApplyView job
StackAdapt15 hours ago

Staff Applied Machine Learning Scientist – Identity

US flagUnited States, +1 more countryFull-timeMachine Learning Engineer$182.4k – $250.8k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers