AI/ML, MLOps Engineer

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Oversee the design and execution of secure, scalable, and maintainable production machine learning architectures.

• Create, sustain, and manage training, validation, and testing datasets.

• Develop repeatable processes for dataset ingestion, preparation, transformation, labeling, validation, versioning, and approval.

• Implement controls to ensure dataset quality, completeness, integrity, provenance, lineage, security, and authorized usage.

• Design and automate comprehensive ML pipelines that encompass preparation, training, evaluation, packaging, deployment, monitoring, retraining, and retirement.

• Deploy machine learning models in approved cloud, application, analytical, and operational settings.

• Establish automated procedures for model building, testing, validation, promotion, deployment, rollback, and recovery.

• Implement model registries and controlled repositories for artifacts, dataset versions, code, configurations, dependencies, evaluation results, and approvals.

• Develop automated testing for model pipelines, data transformations, interfaces, deployment packages, and production behavior.

• Define frameworks for model evaluation, performance thresholds, validation criteria, and release-readiness requirements.

• Assess model accuracy, reliability, robustness, performance, scalability, security, and alignment with mission criteria.

• Implement production monitoring for model performance, data drift, concept drift, inference quality, latency, availability, errors, and resource use.

• Set thresholds, alerts, escalation paths, and response strategies for degradation, drift, data-quality problems, and operational failures.

• Investigate anomalies in models and pipelines, coordinating corrective actions with data, application, infrastructure, security, and mission teams.

• Manage updates to production models, datasets, features, pipelines, configurations, and dependencies through documented review and approval processes.

• Ensure traceability among source datasets, transformations, model versions, evaluation results, deployment records, and production outcomes.

• Develop procedures for rollback and recovery for models, datasets, pipelines, and supporting services.

• Support secure data pipelines for AI development, model evaluation, deployment, monitoring, and operational decision support.

• Implement model and data governance controls, including version control, environment separation, approval gates, auditability, and change management.

• Maintain audit-ready documentation covering model purpose, training data, evaluation methods, limitations, dependencies, approvals, changes, performance, and known risks.

• Assist in evaluating and auditing model outputs to identify inaccurate, unsupported, or unexpected results.

• Contribute to establishing human-review checkpoints and escalation procedures for outputs that could significantly impact mission or operational decisions.

• Ensure compliance with Government data-handling, cloud-security, cybersecurity, and operational-risk regulations.

• Integrate workflows with continuous integration, continuous delivery, automated testing, infrastructure-as-code, monitoring, and incident-management capabilities.

• Collaborate with cybersecurity teams regarding vulnerabilities, access controls, secrets management, software dependencies, logging, encryption, and authorization.

• Coordinate with data warehouse and data engineering teams on data models, metadata, ETL processes, data quality, and governed access.

• Provide technical leadership, architectural guidance, code and design reviews, troubleshooting support, and knowledge transfer.

• Prepare architecture diagrams, lifecycle documentation, deployment procedures, monitoring plans, test results, operating guides, and technical reports.

• Communicate model performance, drift, risks, incidents, corrective actions, and deployment status to technical teams, Government stakeholders, and program leadership.

⛳️ Requirements

• Active Secret clearance or higher.

• Bachelor's degree in computer science, data science, mathematics, engineering, or a related field, or an equivalent credential.

• At least 12 years of relevant professional experience.

• Proven experience leading the architecture of production machine learning systems.

• Familiarity with designing and operating production ML or MLOps environments.

• Experience in developing and maintaining training, validation, and testing datasets.

• Knowledge in establishing model-evaluation methods, performance metrics, validation criteria, and acceptance thresholds.

• Experience in deploying machine learning models in production settings.

• Expertise in automating model build, test, deployment, promotion, rollback, or retraining processes.

• Experience implementing monitoring for model and data drift.

• Understanding of model governance and data governance protocols.

• Proficient in controlling updates for models, datasets, pipelines, and configurations.

• Experience with cloud-based machine learning platforms or production cloud architectures.

• Familiarity with versioning for models, datasets, code, artifacts, and configurations.

• Experience in building or integrating automated testing, continuous integration, and continuous delivery capabilities for machine learning systems.

• Proven ability to monitor production model performance, reliability, and operational health.

• Understanding of secure data pipelines, access controls, audit logging, encryption, secrets management, and software supply-chain security.

• Ability to establish traceability across datasets, transformations, model versions, evaluation results, deployments, and operational outcomes.

• Proficient in documenting and communicating model limitations, performance, drift, risks, incidents, and corrective actions.

• CompTIA Security+ CE or another approved IAT Level II certification.

• Preferred: AWS Certified Machine Learning Engineer - Associate or comparable machine learning platform certification.

• Preferred: Advanced certification in AWS, Microsoft Azure, Google Cloud, Databricks, or a comparable cloud architecture or data engineering certification.

• Preferred: Experience supporting federal or Department of Defense programs.

• Preferred: Experience operating machine learning capabilities in regulated, controlled, or mission-critical environments.

• Preferred: Familiarity with platforms such as MLflow, Amazon SageMaker, Azure Machine Learning, Google Vertex AI, Databricks, Kubeflow, or similar.

• Preferred: Experience with containerization, Kubernetes, infrastructure as code, and cloud-native deployment patterns.

• Preferred: Experience implementing feature stores, model registries, metadata repositories, experiment tracking, or data-lineage tools.

• Preferred: Familiarity with observability platforms, automated alerting, incident response, and production reliability engineering.

• Preferred: Understanding of responsible AI, explainability, output validation, human oversight, bias evaluation, and model-risk management practices.

• Preferred: Knowledge of DoD Impact Level 4 or comparable Government cloud and data-protection requirements.

• Preferred: Experience with analytical models, generative AI, retrieval workflows, or agentic AI systems.

• Preferred: Experience leading technical teams, architecture reviews, or cross-functional ML delivery initiatives.

🏝️ Benefits

• Equal Employment Opportunity and Affirmative Action employer.

• Pay transparency protections.

• Opportunity to be considered for other available positions requiring comparable skills, education, and experience.