Staff Software Engineer, Inference Routing

Posted 12 hours ago

This is a fully remote position, open to applicants in Canada.

πŸ“‹ Description

β€’ Take ownership of the routing layer responsible for determining which GPU handles each request.

β€’ Conceptualize and develop the router and placement logic.

β€’ Construct, manage, and establish the technical trajectory of the routing system in a production environment.

β€’ Ensure that requests are directed to healthy, accessible, and appropriately located GPUs.

β€’ Manage fluctuating capacity and health across various sites.

β€’ Implement robust failure handling to ensure seamless traffic redirection in the event of site or server outages.

β€’ Collaborate closely with engineers managing the network and GPU locations.

β€’ Assist in recruiting engineers for the team.


⛳️ Requirements

β€’ Extensive understanding of the underlying mechanisms of networks, proxies, and load balancers, gained from building or operating production systems.

β€’ Solid foundation in distributed systems concepts: consistency, failure handling, retries, and backpressure.

β€’ Experience in production coding with a systems programming language such as Go or Rust.

β€’ Proficiency in measuring tail latency and diagnosing its root causes.

β€’ Background in being on call for systems that you have developed.

β€’ Experience in establishing technical direction that other engineers can build upon.

β€’ Strong writing skills.

β€’ Proficient use of AI tools, with the ability to verify their output.

β€’ GPU or machine learning experience is not mandatory.


🏝️ Benefits

β€’ Comprehensive health benefits.

β€’ Flexible work hours.

β€’ Opportunity for professional growth and development.

β€’ Collaborative and innovative work environment.

People also viewed

Kismet10 hours ago

Senior Software Engineer, Full Stack

AU flagAustralia OnlyFull-timeFull-stack Engineer
ApplyView job
Kestra10 hours ago

Integrations Software Engineer

IN flagIndia OnlyFull-timeFull-stack Engineer
ApplyView job
The College Board10 hours ago

Principal Engineer

US flagUnited States OnlyFull-timeFull-stack Engineer$200k – $235k/year
ApplyView job
Magnet Talent10 hours ago

Full-Stack Engineer – Saudi Arabian Fintech Startup

SA flagSaudi Arabia OnlyFull-timeFull-stack Engineer
ApplyView job
LawnStarter10 hours ago

Staff Software Engineer, Product

MX flagMexico, +3 more countriesFull-timeFull-stack Engineer$85k – $125k/year
ApplyView job
AeroVect10 hours ago

Software Engineer – Simulation

North AmericaFull-timeFull-stack Engineer$110k – $170k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers