Senior AI Inference Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Assess model placement, parallelism, precision, and memory strategies for single-GPU, multi-GPU, and multi-node setups

• Choose and evaluate serving runtimes such as vLLM, SGLang, TensorRT-LLM, and their successors

• Set and justify concurrency limits, latency and TTFT targets, and throughput commitments

• Create benchmarking methodologies, harnesses, and reproducible measurement practices

• Take ownership of the production model lifecycle, covering loading, startup, health, upgrades, capacity, and failure management

• Design approaches for both multi-tenant and dedicated serving

• Investigate and resolve inference incidents related to serving runtimes, GPUs, drivers, hosts, and networks

• Transform inference expertise into auditable, tested, and automated software solutions

• Deploy, benchmark, debug, tune, automate, and manage real inference systems in production environments

• Assist in automating decisions regarding hardware fit, precision, model splitting, serving runtime, concurrency, and latency-throughput trade-offs

• Engage in technical decision-making and share responsibilities for production and on-call support

⛳️ Requirements

• Demonstrated production experience with large language models under real-world load

• Experience in serving models across multiple GPUs and nodes

• Proficiency in sizing models relative to hardware capabilities

• Knowledge of selecting quantization and precision strategies considering quality and cost implications

• Experience in tuning batching and concurrency settings

• Ability to measure and advocate for latency, TTFT, throughput, utilization, and cost metrics

• Familiarity with NVIDIA and/or AMD inference environments

• Production-level Python development skills

• Competence in Linux and system debugging below the container level

• Experience in diagnosing inference failures beyond the serving runtime

• Understanding of distributed inference and multi-node networking concepts

• Background in benchmarking and comparing serving runtimes

• Experience in automating deployment decisions or developing schedulers, placement systems, capacity planners, or similar infrastructure

• Capability to work across design, implementation, debugging, and operational tasks

• Willingness to participate in production on-call responsibilities

• Readiness to thrive in a fast-paced, uncertain early-stage startup environment

🏝️ Benefits

• Meaningful equity

• Remote and distributed work opportunities across various time zones

• Employment or contract engagement options based on location

• Direct access to company founders

• Shared responsibilities for production and on-call support

• Involvement in technical decision-making processes