
Machine Learning Engineer, Evaluation, Reliability
Posted 4 days ago

Posted 4 days ago
This is a fully remote position, open to applicants in United States.
• Develop and uphold evaluation systems for AI-driven applications at Bask Health.
• Create automated testing frameworks for extensive language models and AI application workflows.
• Generate and sustain representative datasets for assessing model performance.
• Establish benchmarks that evaluate response accuracy, relevance, completeness, and consistency.
• Design evaluation pipelines for prompts, retrieval systems, and model configurations.
• Assess AI assistants and agents through realistic user scenarios and workflow requirements.
• Construct automated regression tests to identify performance changes across releases.
• Analyze hallucination rates, unsupported claims, and other quality failures of models.
• Assess retrieval-augmented generation systems for document relevance and answer grounding.
• Formulate tests for structured model outputs and schema compliance.
• Evaluate tool-calling accuracy and the selection of agent actions.
• Examine AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.
• Conduct evaluation checks for sensitive patient and operational information.
• Develop dashboards and reports that convey AI quality metrics to engineering teams.
• Analyze evaluation outcomes and recognize recurring failure patterns.
• Investigate production incidents related to inaccurate or unreliable AI outputs.
• Collaborate with applied AI engineers to enhance prompts, retrieval logic, and application functionality.
• Assist in model comparisons and controlled experiments.
• Assess trade-offs among model quality, latency, reliability, and inference costs.
• Create automated quality checks for CI/CD and AI feature release workflows.
• Implement monitoring and alerting systems for production AI performance.
• Develop tools for reviewing, labeling, and analyzing model responses.
• Support the maintenance of evaluation datasets as products and user behaviors evolve.
• Document evaluation methodologies, test cases, and known limitations of models.
• Collaborate with product teams to establish measurable acceptance criteria for AI features.
• Work alongside senior engineers on evaluation standards and improvements in reliability.
• Engage in code reviews, technical discussions, and engineering planning.
• Take ownership of designated evaluation and reliability projects from implementation to deployment and continuous enhancement.
• Over 3 years of professional experience in machine learning engineering, software engineering, applied AI, or a related technical discipline.
• Proven experience in building and maintaining production software or machine learning applications.
• Proficient programming skills in Python, TypeScript, or similar languages.
• Knowledge of large language models and contemporary generative AI applications.
• Experience in testing or evaluating machine learning models or AI-driven software.
• Familiarity with machine learning evaluation metrics and experimental methodologies.
• Experience in developing automated tests and validation pipelines.
• Understanding of prompt engineering and analysis of model outputs.
• Ability to analyze both structured and unstructured datasets.
• Proven experience in scripting or creating services for data processing and evaluation.
• Knowledge of model accuracy, precision, recall, and other applicable performance measures.
• Familiarity with API integrations and backend application development.
• Experience with relational databases or analytical data systems.
• Ability to recognize patterns in model failures and convert findings into engineering enhancements.
• Understanding of regression testing and software quality assurance practices.
• Familiarity with logging, monitoring, and troubleshooting in production environments.
• Experience utilizing Git and collaborative software development methodologies.
• Awareness of the limitations and failure modes associated with large language models.
• Ability to design representative test cases and evaluation scenarios.
• Familiarity with cloud-based development and deployment platforms.
• Proven ability to write maintainable, well-tested production code.
• Strong analytical skills and attention to detail.
• Ability to communicate technical evaluation results effectively.
• Experience working collaboratively with engineers, product managers, and cross-functional teams.
• Capability to independently manage assigned technical projects.
• Competitive salary and performance-based bonuses.
• Comprehensive health insurance plans.
• Flexible work hours and remote work options.
• Opportunities for professional development and continuing education.
• Engaging and inclusive work environment.
Memory Bridge
Entrada
Find Your Talent (FYT)
Inmetrics
Get handpicked remote jobs straight to your inbox weekly.