Senior ML Operations Engineer – Ads Platform

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the entire model lifecycle from experimentation to validation, deployment, versioning, retraining, rollback, and retirement.

• Define criteria for model promotion, establish release processes, and create rollback strategies.

• Design and implement scalable machine learning data and feature engineering pipelines utilizing Spark/PySpark, S3, and Apache Iceberg.

• Create incremental data processing pipelines that ensure data quality validation, lineage tracking, point-in-time accuracy, and consistency between training and serving features.

• Develop monitoring systems for data quality, drift detection, prediction accuracy, service reliability, latency, and infrastructure costs.

• Collaborate with data scientists to establish evaluation metrics for both offline and production environments, including delayed outcome labels.

• Leverage production data to enhance model performance, platform reliability, deployment speed, and overall infrastructure efficiency.

• Construct reusable training, evaluation, and deployment pipelines applicable to various models and approaches.

• Develop orchestration workflows that manage dependencies, scheduling, retries, backfills, and failure recovery processes.

• Transform scientific requirements into sustainable production systems with well-defined interfaces, ownership, and operational expectations.

• Create and maintain an accessible model hub featuring documentation, evaluation results, release history, versioning, and monitoring tools.

• Establish tested CI/CD workflows, reproducible environments, actionable alerts, incident-response protocols, and operational runbooks.

• Set MLOps priorities in collaboration with Data Science and Engineering teams, proactively identifying risks before they escalate into production issues.

• Develop reusable engineering standards, mentor fellow team members, and alleviate the operational load on data scientists.

⛳️ Requirements

• Minimum of 5 years of experience in constructing and managing production ML systems beyond initial deployment, including monitoring, retraining, incident response, and continuous enhancement.

• Proficient in Python with strong production software development skills and a solid understanding of SQL.

• Practical experience with Spark/PySpark for large-scale distributed data processing, encompassing feature engineering, historical aggregations, incremental processing, and the creation of training datasets.

• Familiarity with modern data lake or lakehouse architectures, such as S3, Apache Iceberg, or similar technologies.

• Experience utilizing a workflow orchestration engine for scheduling, dependency management, retries, backfills, and recovery tasks.

• Hands-on experience in building infrastructure using Terraform.

• Experience in deploying and managing containerized production workloads on Kubernetes.

• Proven ability to build reusable training, evaluation, or deployment infrastructure utilized by numerous models, scientists, or teams.

• Experience in implementing model and dataset versioning, experiment tracking, lineage, and reproducible release processes.

• Capability to differentiate between service or infrastructure failures and changes in model effectiveness.

• Experience in creating CI/CD pipelines, monitoring systems, alerting mechanisms, rollback strategies, and recovery procedures for production ML environments.

• Demonstrated skill in balancing model quality, delivery speed, reliability, and infrastructure costs.

• Strong cross-functional communication skills and experience collaborating with data scientists to maintain scientific integrity as models transition to production.

• Ability to articulate a production incident or model degradation issue, including detection methods and subsequent improvements.

• Preferred: experience in developing platform services, infrastructure, or internal tools using Go.

• Preferred: experience managing or owning a model registry, model hub, or similar system.

• Preferred: experience in supporting ML systems featuring delayed outcome labels.

• Preferred: a history of enhancing deployment speed, model quality, system reliability, infrastructure costs, or reducing manual operational effort.

• Preferred: experience in designing configurable ML infrastructure that supports various modeling frameworks or methodologies.

• Preferred: evidence of reusable infrastructure, standards, or documentation being adopted across multiple teams.

• Preferred: experience mentoring engineers or data scientists and influencing technical priorities across an ML portfolio.

• Preferred: experience handling sensitive information with appropriate access controls.

🏝️ Benefits

• Equity for full-time employees.

• 401(k) matching dollar-for-dollar up to 4% of eligible pay.

• Comprehensive medical, dental, and vision plans, including coverage for pets.

• Up to $10,000 per year in education reimbursement.

• Participation in Employee Resource Groups and Inclusion Council.

• Flexible Paid Time Off (PTO).

• 9 paid holidays per year.

• A week-long break at the end of the year.

• 14 weeks of fully paid parental leave.

• 6 additional weeks of paid childbirth recovery leave for birthing parents.

• Flexible return-to-work schedule.

• One-time $2,000 Calvin Care Cash incentive for new family members.

• Flexible work environment.

• Hardware and software provided for remote work.