Principal ML Performance Engineer – GPU Optimization

This is a fully remote position, open to applicants in New York.

📋 Description

• Profile and enhance training and inference processes for structural and generative models, including transformers, diffusion models, and geometric deep learning techniques.

• Develop and fine-tune custom kernels utilizing CUDA and Triton.

• Leverage compilers such as torch.compile, TensorRT, and XLA whenever advantageous.

• Scale distributed training across 32–64 nodes through FSDP, DeepSpeed, tensor parallelism, pipeline parallelism, and mixed precision methodologies.

• Decrease inference costs by optimizing memory scaling for large complexes, enhancing diffusion sampling efficiency, batching ragged inputs, and maximizing throughput across up to 1000 GPUs.

• Oversee GPU cluster efficiency on GCP, concentrating on scheduling, utilization, spot strategy, and cost analysis.

• Create benchmarks and profiling tools for the research team.

• Contribute to the development of Proxima’s AI and data-generation platform aimed at proximity therapeutics and protein-interaction discovery.

⛳️ Requirements

• A minimum of 6+ years of experience in ML systems, HPC, or performance engineering.

• BS/MS/PhD in Computer Science, Electrical Engineering, or a related discipline.

• Ability to establish technical direction beyond coding, select infrastructure, influence research teams, and mentor engineers.

• In-depth knowledge of PyTorch internals with practical experience in profiling and resolving real bottlenecks.

• Experience working with CUDA and Triton.

• Proficient in interpreting Nsight output.

• Strong comprehension of memory bandwidth and occupancy principles.

• Experience in distributed training at a multi-node scale.

• High proficiency in Python and C++.

• Capable of identifying a model they have significantly accelerated and quantifying the performance improvement.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and remote work options.

• Opportunity for continuous learning and professional development.

• Collaborative and innovative work environment.