
Staff Software Engineer, Inference Routing
Posted 12 hours ago

Posted 12 hours ago
This is a fully remote position, open to applicants in Canada.
β’ Take ownership of the routing layer responsible for determining which GPU handles each request.
β’ Conceptualize and develop the router and placement logic.
β’ Construct, manage, and establish the technical trajectory of the routing system in a production environment.
β’ Ensure that requests are directed to healthy, accessible, and appropriately located GPUs.
β’ Manage fluctuating capacity and health across various sites.
β’ Implement robust failure handling to ensure seamless traffic redirection in the event of site or server outages.
β’ Collaborate closely with engineers managing the network and GPU locations.
β’ Assist in recruiting engineers for the team.
β’ Extensive understanding of the underlying mechanisms of networks, proxies, and load balancers, gained from building or operating production systems.
β’ Solid foundation in distributed systems concepts: consistency, failure handling, retries, and backpressure.
β’ Experience in production coding with a systems programming language such as Go or Rust.
β’ Proficiency in measuring tail latency and diagnosing its root causes.
β’ Background in being on call for systems that you have developed.
β’ Experience in establishing technical direction that other engineers can build upon.
β’ Strong writing skills.
β’ Proficient use of AI tools, with the ability to verify their output.
β’ GPU or machine learning experience is not mandatory.
β’ Comprehensive health benefits.
β’ Flexible work hours.
β’ Opportunity for professional growth and development.
β’ Collaborative and innovative work environment.
Kismet
Kestra
The College Board
Magnet Talent
Get handpicked remote jobs straight to your inbox weekly.