Senior Compiler Engineer – AI

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Design and execute program transformations on JAX programs (jaxpr) for extensive training, which includes pipeline scheduling, task placement, sharding, and overlapping communication/computation.

• Create optimizing compiler and runtime technologies aimed at enhancing deep learning training on large-scale distributed systems.

• Compile graph intermediate representations into target-specific code.

• Develop distributed runtimes that allocate per-stage programs across multiple GPUs and facilitate data transfer between stages using communication libraries such as NCCL.

• Propel compiler technology for high-performance large-scale training tasks, including mixture-of-experts models.

• Collaborate with teams from JAX, XLA, and hardware architecture.

• Integrate JaxPP into JAX training frameworks, such as MaxText.

• Contribute to the open-source JaxPP initiative through maintenance, testing, performance monitoring, and support for adoption.

⛳️ Requirements

• A Master's or PhD in Computer Science, Computer Engineering, or a related field, or equivalent experience.

• Over 6 years of pertinent work or research experience in compilers, distributed systems, or deep learning training.

• Proficient in compiler design and implementation.

• Experience in designing, developing, and deploying distributed systems.

• Familiarity with deep learning frameworks (such as JAX and PyTorch).

• Expertise in GPU training and inference of deep learning models.

• Proficient in C/C++ and Python programming languages.

• Strong software design capabilities, including debugging, performance evaluation, and test design.

• Experience with parallel computing libraries (such as MPI).

• Ability to work autonomously, define project goals and scope, and lead development initiatives.

• Excellent communication skills.

• Familiarity with JAX internals, XLA, StableHLO, or MLIR is a plus.

• Experience in building or optimizing large training jobs using pipeline, tensor, expert, and data parallelism is a plus.

• Knowledge of GPU communication libraries such as NCCL, NVSHMEM, or RDMA, and CPU and/or GPU architecture is advantageous.

• Experience with CUDA programming is an asset.

• Contributions to open-source compiler or machine learning projects are a plus.

• Experience mentoring early-career engineers and interns is an added bonus.

🏝️ Benefits

• Equity

• Benefits package

• Competitive salary