Senior Staff Machine Learning Systems Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the technical framework for evaluation systems, encompassing metrics, judges, scorers, statistical methods, and failure-tracking infrastructure.

• Design and expand ingestion, transformation, dataset-versioning, labeling, and calibration pipelines.

• Tackle scaling and complexity challenges in AI evaluation.

• Define evaluation strategies for new AI services.

• Spearhead multi-team initiatives to replace manual reviews with statistically robust automated gates.

• Oversee adversarial and red-team evaluation programs, test suites, and failure taxonomies.

• Resolve technical disagreements across teams and foster alignment among engineering, product, and AI leadership.

• Create reusable evaluation methodologies and standards.

• Navigate ambiguous cross-team issues from conception to fully specified production systems.

• Lead significant platform enhancements and modernization of core systems.

• Cultivate relationships across ML engineering, data science, platform engineering, clinical, legal, and product teams.

• Disseminate evaluation and data-pipeline knowledge through presentations and written documentation.

• Mentor engineers, including those with extensive experience and seniority.

⛳️ Requirements

• Over 10 years of experience in ML infrastructure, data engineering, or evaluation/testing systems.

• Practical experience in designing and calibrating LLM judges and scorers.

• Proficient in developing statistically sound regression-testing methodologies, including paired significance testing and adjustments for multiple comparisons.

• Experience in measuring agreement with human labels.

• Background in designing adversarial and red-team evaluation methodologies.

• Hands-on experience with dataset versioning, feature and benchmark pipelines, labeling and calibration workflows, and high-throughput ingestion and transformation systems.

• Proven ability to create standards adopted by wider groups.

• Experience in leading multi-team projects to successful completion and addressing technical disagreements.

• History of mentoring engineers, including those at senior levels.

• Excellent communication skills with the ability to adapt technical concepts for various audiences.

• Strong proficiency in Python.

• Statistical fluency adequate for designing and defending production testing frameworks.

• Familiarity with Databricks, MLflow, Unity Catalog, or similar platforms is preferred.

• Experience in developing reporting tools for non-technical teams is preferred.

• Previous experience in regulated industries is preferred.

• Contributions to company-wide talks or write-ups that have influenced other teams are preferred.

• Must be legally authorized to work in the U.S. without restrictions for any employer.

• Must not require current or future immigration sponsorship by Hims & Hers to work in the U.S.

🏝️ Benefits

• Competitive salary and equity compensation for full-time positions.

• Unlimited PTO.

• Company holidays.

• Quarterly mental health days.

• Comprehensive medical, dental, and vision benefits.

• Parental leave.

• Employee Stock Purchase Program (ESPP).

• 401k benefits with employer matching contributions.

• Offsite team retreats.

• Flexible/remote work options.

• Reasonable accommodations for qualified individuals with disabilities and disabled veterans.