Senior ML Operations Engineer – Monetization

This is a fully remote position, open to applicants in United States.

📋 Description

• Manage the entire model lifecycle, from experimentation and validation to deployment, version control, retraining, rollback, and eventual retirement.

• Set criteria for model promotion, establish release processes, and define rollback strategies.

• Develop monitoring systems for data quality, drift, prediction accuracy, service reliability, latency, and infrastructure costs.

• Collaborate with data scientists to identify offline and production evaluation metrics, including use cases with delayed labels.

• Leverage production data to enhance model performance, platform reliability, deployment efficiency, and infrastructure effectiveness.

• Create reusable training, evaluation, and deployment pipelines applicable across various models and methodologies.

• Design orchestration workflows for managing dependencies, scheduling tasks, handling retries, backfilling data, and recovering from failures.

• Convert scientific requirements into sustainable production systems with well-defined interfaces, ownership, and operational expectations.

• Establish and maintain a model hub that is easily accessible for documentation, evaluation outcomes, release history, version control, and monitoring.

• Implement robust CI/CD workflows, reproducible environments, actionable alerts, incident response protocols, and operational runbooks.

• Collaborate with Data Science and Engineering teams to set MLOps priorities and proactively identify risks prior to production issues.

• Set engineering best practices, mentor team members, and alleviate the operational load on data scientists.

⛳️ Requirements

• Over 5 years of experience managing production ML systems beyond initial launch, including monitoring, retraining, incident response, and ongoing enhancement.

• Proficient in production software development using Python and skilled in SQL.

• Practical experience with Spark for distributed data processing.

• Familiarity with a workflow orchestration engine for scheduling, managing dependencies, handling retries, backfilling, and recovery.

• Direct experience building infrastructure using Terraform.

• Background in deploying and managing containerized production workloads with Kubernetes.

• Proven experience in creating reusable training, evaluation, or deployment infrastructure used by multiple models, scientists, or teams.

• Experience in implementing model and dataset versioning, experiment tracking, lineage, and reproducible releases.

• Ability to differentiate between service or infrastructure failures and changes in model performance.

• Experience in constructing CI/CD pipelines, monitoring systems, alerting, rollback solutions, and recovery procedures for production ML systems.

• Demonstrated capability to balance model quality, delivery speed, reliability, and infrastructure costs.

• Strong cross-functional communication skills and experience working with data scientists to maintain scientific integrity as models transition to production.

• Ability to articulate production incidents or model performance degradation, detailing detection methods and subsequent improvements.

• Preferred: Experience in developing platform services, infrastructure, or internal tools using Go.

• Preferred: Background in building or managing a model registry, model hub, or similar system.

• Preferred: Experience in supporting ML systems that utilize delayed outcome labels or quality metrics.

• Preferred: Proven track record of enhancing deployment speed, model quality, system reliability, infrastructure costs, or reducing manual operational efforts.

• Preferred: Experience designing configurable ML infrastructure supporting various modeling frameworks or techniques.

• Preferred: Evidence of creating reusable infrastructure, standards, or documentation adopted across multiple teams.

• Preferred: Experience mentoring engineers or data scientists and shaping technical priorities.

• Preferred: Familiarity with handling sensitive information and establishing appropriate access boundaries.

🏝️ Benefits

• Equity offered to full-time employees.

• 401(k) matching at a dollar-for-dollar rate up to 4% of eligible pay.

• Comprehensive medical, dental, and vision insurance plans, inclusive of pets.

• Annual education reimbursement of $10,000.

• Employee Resource Groups available for support.

• Flexible paid time off (PTO) policy.

• Nine paid holidays each year.

• A week-long break at the end of the year.

• Fourteen weeks of fully paid parental leave.

• Six additional weeks of paid recovery leave for birthing parents.

• Flexible return-to-work arrangements.

• One-time $2,000 Calvin Care Cash incentive for employees welcoming new family members.

• Flexible work environment options.

• Provision of hardware and software for remote work.