📋 Description

• Design and conduct experiments aimed at enhancing agent quality: improved tool utilization, enhanced reasoning, and superior outputs, utilizing frameworks such as DSPy and VLLM.

• Develop and sustain evaluation infrastructure to assess effectiveness and identify regressions before they impact customers.

• Optimize LLM inference concerning latency, costs, model routing, and quality trade-offs.

• Collaborate with product teams on model selection and performance assessment benchmarks.

• Work in close partnership with product engineers and PMs to convert customer quality challenges into ML hypotheses and viable solutions.

• Take ownership of models throughout their lifecycle: from research and experimentation to deployment in production.

⛳️ Requirements

• A minimum of 5 years of experience in a specialized ML Engineer position (distinct from ML-adjacent software engineering).

• Proficient in Python; experience with PyTorch, VLLM, DSPy, or analogous LLM optimization frameworks.

• Practical experience with large language models in a production setting: including prompt engineering, fine-tuning, evaluation, and inference optimization.

• Capability to transition from vague problems ('this agent isn't performing well') to experimental design and delivered improvements.

• Comfort in working with minimal supervision within an early-stage product environment.