Director, AI Engineering

This is a fully remote position, open to applicants in California, +3 more states.

📋 Description

• Collaborate with engineers, research scientists, technical program managers, and product managers to create AI-driven products.

• Manage the design, development, testing, deployment, and operation of distributed training, fine-tuning, reinforcement learning, GPU scheduling, fault tolerance, utilization, and model experimentation systems.

• Make build-versus-buy assessments regarding open-source and SaaS AI technologies such as AWS Ultraclusters, Hugging Face, vector databases, and PyTorch.

• Implement techniques that enhance scalability, cost efficiency, throughput, and reliability in large-scale distributed training and fine-tuning.

• Oversee GPU capacity planning and cost management across various teams.

• Contribute to the technical vision and long-term strategy for foundational AI systems.

• Attract, maintain, mentor, and develop AI engineering talent.

• Convert enterprise AI strategy into actionable portfolio-level execution plans across different product areas.

• Expand AI engineering practices through shared infrastructure, reusable components, observability, and governance frameworks.

• Set enterprise Responsible AI standards that encompass fairness metrics, model evaluation, documentation, and audit readiness.

• Collaborate with research, compliance, and enterprise risk teams to uphold ethical and regulatory standards.

⛳️ Requirements

• A Bachelor's degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or related fields with at least 8 years of experience in developing AI and ML algorithms or technologies; or a Master's degree in these fields with at least 6 years of such experience.

• A minimum of 3 years of experience in people leadership.

• Over 5 years of experience managing and leading an engineering team (preferred).

• More than 7 years of experience in building and operating large-scale ML or GPU training infrastructure on cloud platforms (preferred).

• Practical experience with distributed training at scale, including multi-node, multi-GPU jobs, and parallelism strategies such as PyTorch FSDP, DeepSpeed, and Megatron.

• Proficiency in programming languages such as Python, Go, C++, or CUDA.

• Familiarity with ML orchestration and scheduling tools like Kubernetes, Kubeflow, Kueue, Slurm, Ray, KServe, and vLLM.

• Experience in operating large GPU fleets with a focus on reliability, fault tolerance, utilization, and cost-effectiveness.

• Experience in right-sizing GPU clusters, instance types, interconnect, and quotas.

• A strong interest in current AI and ML systems research and the application of innovative training and optimization techniques.

• Exceptional communication and presentation skills.

• Experience in building and leading a multi-team AI organization.

• Capability to execute long-term AI platform strategies that align with enterprise priorities and regulatory frameworks.

• Experience in establishing cross-functional operating rhythms and review cadences.

• Capital One will consider sponsoring a new qualified applicant for employment authorization.

🏝️ Benefits

• Performance-based incentive compensation, which may include cash bonuses and/or long-term incentives (LTI).

• A comprehensive, competitive, and inclusive range of health, financial, and other benefits that support overall well-being.

• Employment authorization sponsorship may be considered for a new qualified applicant.

• Reasonable accommodations for applicants with disabilities.