This is a fully remote position, open to applicants in United States.

📋 Description

• Develop a caching layer to prevent direct external system calls from every agent tool.

• Implement event streaming to reduce reliance on synchronous HTTP chains between services.

• Enhance agent dispatch by incorporating health checks, circuit breakers, retry logic, and observability to differentiate between failure and degradation.

• Establish platform observability through service-level metrics, cross-service tracing, and proactive alerting.

• Participate in orchestration migration and infrastructure-as-code initiatives.

• Refine deployment methodologies using canary releases, traffic shifting, and dependable rollback processes.

• Collaborate with the Principal Engineer.

• Report to the Chief AI Officer, and subsequently to the Senior Engineering Manager.

⛳️ Requirements

• Hands-on experience in ensuring the reliability of distributed systems, covering caching, queuing, circuit breaking, backpressure, timeouts, and idempotency.

• Proficiency in Python and familiarity with containerized environments.

• Practical knowledge of Kubernetes or a similar orchestration tool.

• Experience with event streaming and understanding its associated failure modes.

• Expertise in observability, focusing on metrics and distributed tracing.

• Ability to identify and alert on symptoms rather than underlying causes.

• Proven track record of delivering reliability enhancements with measurable outcomes.

• Preferred: Experience with durable workflow orchestration.

• Preferred: Knowledge of blue/green and canary deployment practices at scale.

• Preferred: Skills in performance profiling of Python services.