This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Develop and assess routing policies for LLM traffic to optimize the use of a variety of model systems.

• Construct and execute agentic benchmarks (e.g., Terminal-Bench) to evaluate algorithm performance, transforming results into calibration data and routing profiles.

• Contribute to an open-source repository by providing design documentation, participating in code reviews, and making community contributions.

• Work collaboratively with engineering teams across NVIDIA to guarantee seamless integration of our software throughout the NVIDIA accelerated serving stack.

⛳️ Requirements

• Bachelor's or Master's degree in Computer Science or equivalent experience.

• Over 8 years of industry experience with Deep Learning frameworks (PyTorch or TensorFlow).

• Proven experience in designing or conducting LLM evaluations/benchmarks—preferably agentic ones—and deriving statistically valid conclusions from them.

• Familiarity with contemporary techniques in Machine Learning, Deep Neural Networks, Natural Language Processing, or Speech Recognition.

• An empirical research approach: developing hypotheses about new algorithms, performing calibrations, and refining results.

• Excellent communication and interpersonal skills, along with the capability to thrive in a dynamic and distributed team environment.

• A background in mentoring junior engineers and interns is highly advantageous.

• A commitment to continuous growth and learning.

• Strong fundamentals in computer science—encompassing algorithms and data structures, computational complexity, parallel and distributed computing, and system software.