Senior Machine Learning Engineer, Defensive Agent

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and manage training and post-training pipelines, which encompass data preparation, fine-tuning, preference optimization, experiment tracking, artifact management, and ensuring reproducibility.

• Create the inference and serving layer featuring provider routing, fallback mechanisms, regional data residency pinning, batching, and caching.

• Oversee the model-artifact release process, which includes versioned prompts, model selections, tool definitions, shadow deployments, canary deployments, and rollback procedures.

• Establish model-layer monitoring for behavioral drift, regression detection, output quality, latency, and manage per-tenant token and cost accounting with budget enforcement.

• Construct data and context pipelines for inference, integrating retrieval and embedding infrastructure across attack paths, configurations, and remediation data.

• Ensure complete tenant isolation throughout the process.

• Optimize costs and latency along the inference path while making trade-offs transparent.

• Develop product features in ETL and GraphQL for model outputs, run histories, and evaluation results.

• Collaborate with AI researchers to transition prototypes into production and relay production constraints and failure data back into the research direction.

• Deploy defensive models into production and maintain their operation across thousands of customer tenants.

⛳️ Requirements

• Bachelor's Degree in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.

• Over 5 years of professional software engineering experience.

• Proficient in production-level Python.

• Proven experience in taking ML or LLM-backed systems from prototype to production and managing their operations.

• Practical experience with ML pipelines and tools, including training or fine-tuning workflows, experiment tracking, artifact and model registries, and reproducible data preparation.

• Background in building and operating production-level inference or model-serving infrastructure, with a focus on latency and cost optimization.

• Familiarity with application development on AWS, Azure, or GCP.

• Proficient in Docker and Kubernetes.

• Strong SQL skills with experience in production data pipelines.

• Experience in operationalizing LLM or agentic systems.

• Hands-on post-training experience, including supervised fine-tuning, distillation, preference optimization, or reinforcement learning.

• Experience with model gateways or multi-provider routing, and self-hosted or customer-hosted inference solutions such as vLLM, TGI, or Bedrock.

• Knowledge of GPU infrastructure, quantization, or inference optimization.

• Experience with relational databases such as PostgreSQL and graph databases like Neo4j.

• Familiarity with GraphQL backends.

• Experience with observability tools such as Datadog, Prometheus, or Grafana, as well as distributed tracing.

• Experience in deploying ML solutions in regulated, air-gapped, or customer-controlled environments, or under compliance regimes such as FedRAMP.

• Legally authorized to work in the United States.

• Bachelor's degree or equivalent practical experience.

🏝️ Benefits

• Equity package in the form of stock options for all full-time positions.

• Health, vision, and dental insurance coverage for you and your family.

• Flexible vacation policy.

• Generous parental leave.

• Options for remote and hybrid work models based on role and location.

• Opportunities for career development.

• An inclusive and collaborative work culture.