Staff Software Engineer, AI Gateway

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the technical vision for core AI Gateway systems, including the model router, MCP gateway, and control plane.

• Ensure that architectural choices accommodate a tenfold increase in usage.

• Identify gaps in infrastructure, processes, and tools, and lead initiatives to address these issues in production.

• Define, implement, and deliver the Gateway's technical strategy in collaboration with engineering leadership.

• Tackle complex challenges involving dynamic model selection, auto-failover mechanisms, cost-based routing, and comprehensive policy enforcement.

• Act as the authority on secure credential storage, session summarization, caching, and rate limiting.

• Develop infrastructure for evaluating, replaying, and backtesting the behavior of agents and LLMs.

• Guide senior and mid-level engineers through code reviews, design assessments, and practical examples.

• Collaborate with product, design, and leadership teams on the control plane UI direction and overall product strategy.

• Create a unified abstraction layer across various model and tool providers, including OpenAI, Anthropic, Google, and AWS Bedrock.

• Design secure and scalable identity and credential vaulting solutions for enterprise-level access.

• Establish real-time and historical analytics infrastructure to monitor AI agent behaviors.

• Ensure full policy enforcement and logging from prompt initiation to tool response.

• Set benchmarks for developer experience through user-friendly APIs and high-quality observability.

⛳️ Requirements

• A minimum of 8 years of backend engineering experience.

• Proven track record of managing production-grade systems at a large scale.

• Extensive expertise in distributed systems, cloud-native infrastructure, and API design.

• Experience in developing secure, high-throughput services such as gateways, proxies, load balancers, and policy engines.

• Proficiency in Go and/or Rust programming languages.

• History of identifying systemic issues and implementing solutions across team boundaries.

• Strong product-oriented mindset with experience in shaping technical directions.

• Capability to work independently with considerable freedom.

• Bachelor's degree in Computer Science, Engineering, or a related discipline, or equivalent practical experience.

• Previous exposure to OpenAI, Anthropic, or similar LLM APIs is advantageous.

• Familiarity with RAG architectures, vector databases, or agent frameworks is a plus.

• Experience with policy engines, observability frameworks, or API gateways is a plus.

• Understanding of OAuth 2.1, secret management, and cloud IAM systems is beneficial.

• Experience with Kubernetes, Docker, and microservices architecture is a plus.

• Sponsorship will be considered on a case-by-case basis depending on business needs.

🏝️ Benefits

• Equity for all full-time employees.

• Remote-first work from home policy.

• Flexible scheduling options.

• Generous paid time off (PTO).

• Designated quarterly Whaleness Days.

• Special end-of-year Whaleness break.

• Support for home office setup.

• Technology stipend of US$100 net per month.

• Annual learning and development stipend for conferences, courses, certifications, and ongoing education.

• 16 weeks of paid parental leave after six months of employment.

• Comprehensive medical benefits.

• Retirement benefits.

• Paid holidays.

• Docker swag.