Senior AI/ML Engineer, MLOps

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Oversee the management of training, validation, and testing datasets utilized for the development and assessment of machine learning models.

• Create repeatable methodologies for the ingestion, preparation, transformation, validation, versioning, and controlled promotion of datasets.

• Assess datasets for quality, completeness, consistency, integrity, provenance, and suitability for their intended use.

• Ensure traceability among source data, transformed datasets, features, model versions, evaluation outcomes, and production deployments.

• Design and sustain automated pipelines for data preparation, model training, model evaluation, packaging, deployment, monitoring, and rollback.

• Deploy machine learning models into sanctioned cloud, application, analytical, and operational settings.

• Automate processes related to model building, testing, validation, promotion, deployment, and recovery.

• Set up and maintain model registries and controlled repositories encompassing models, datasets, code, configurations, dependencies, and evaluation results.

• Implement automated unit, integration, regression, data-quality, pipeline, and deployment testing.

• Establish evaluation procedures, performance metrics, validation thresholds, and release-readiness criteria for models.

• Monitor production models for accuracy, performance, availability, latency, errors, data drift, concept drift, and unexpected behavior.

• Configure alerts, dashboards, thresholds, and escalation protocols for model degradation, drift, data-quality issues, and pipeline failures.

• Investigate anomalies in production models, datasets, pipelines, and deployments.

• Coordinate corrective measures with data, application, infrastructure, security, testing, and mission teams.

• Manage updates to production models, datasets, pipelines, features, dependencies, and configurations through documented change and approval processes.

• Maintain version control for model code, training logic, deployment scripts, datasets, features, configurations, and infrastructure definitions.

• Develop and assess rollback procedures for updates related to models, datasets, pipelines, and configurations.

• Support secure data pipelines utilized for model training, evaluation, deployment, monitoring, and operational analysis.

• Keep documentation that details model purpose, datasets, evaluation methods, performance, limitations, dependencies, deployments, changes, incidents, and known risks.

• Assist with auditing and validation of model outputs to identify inaccuracies, unsupported claims, or unexpected results.

• Aid in implementing human-review and escalation checkpoints for model outputs that may influence mission or operational decisions.

• Integrate MLOps workflows with continuous integration, continuous delivery, infrastructure-as-code, automated testing, monitoring, and incident management capabilities.

• Support data and model governance requirements, such as access controls, versioning, approval gates, traceability, change control, and audit logging.

• Collaborate with cybersecurity personnel regarding vulnerabilities, identity and access controls, secrets management, software dependencies, encryption, logging, and authorization requirements.

• Work together with data engineering teams on data ingestion, transformation, metadata, warehouse integration, data quality, and governed access.

• Assist with architecture reviews, code reviews, release-readiness assessments, troubleshooting, operational handoff, and knowledge transfer.

• Prepare lifecycle documentation for models, deployment instructions, rollback procedures, monitoring plans, test results, technical diagrams, and operating guides.

• Communicate the status, performance, drift, risks, incidents, and corrective actions of models to technical teams, Government stakeholders, and program leadership.

⛳️ Requirements

• Possession of an active Secret clearance or higher.

• Bachelor’s degree in computer science, data science, engineering, or a related discipline, or an acceptable equivalent.

• At least 10 years of pertinent professional experience.

• Experience in maintaining machine learning training, validation, and testing datasets.

• Proven experience in building or sustaining automated machine learning deployment pipelines.

• Demonstrated experience in deploying machine learning models within production environments.

• Experience in monitoring production models and supporting related services.

• Proficient in detecting and addressing model or data drift.

• Familiarity with version control for model code, datasets, configurations, pipelines, or deployment artifacts.

• Experience in developing and executing rollback procedures for models, datasets, pipelines, or production configurations.

• Experience in managing model and dataset updates through documented change, testing, and approval procedures.

• Experience in implementing automated testing for machine learning models, data pipelines, APIs, or deployment workflows.

• Knowledge of cloud-based machine learning services or production cloud environments.

• Understanding of model evaluation, validation metrics, production monitoring, model governance, and data governance.

• Ability to maintain traceability across datasets, transformations, model versions, evaluation results, deployments, and production outcomes.

• Capability to investigate and resolve issues related to models, datasets, pipelines, and deployments.

• Understanding of secure data pipelines, access controls, audit logging, encryption, secrets management, and dependency security.

• Proficient in documenting and communicating model performance, drift, limitations, risks, incidents, and corrective actions.

• CompTIA Security+ CE or another approved IAT Level II certification.

🏝️ Benefits

• Regular full-time employment.

• Opportunity for remote work.

• Preference for candidates located in the Chicago area.

• Chance to contribute to a USMEPCOM ENLIST government program.

• Protections under Equal Employment Opportunity and Affirmative Action.