Technical Staff Member – Observability, Reliability

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Enhance the observability framework for logs, metrics, traces, and alerting.

• Ensure that every cloud and on-premise data plane reports its active release status, health, heartbeat, logs, metrics, and usage to the control plane.

• Integrate telemetry within customer clusters utilizing outbound-only agent connections.

• Identify discrepancies between the desired state and the actual state in each environment.

• Oversee the health of deployment and runtime agents.

• Provide insights into transient workloads, such as Ray clusters executing batch inference.

• Establish Service Level Objectives (SLOs), lead incident response and postmortems, and work towards reducing Mean Time to Recovery (MTTR), including customer-coordinated fixes.

• Lower telemetry expenses by removing redundant data and enhancing signal quality.

• Attain 99.9% serving availability, decrease incidents and MTTR, minimize state drift, and ensure all agents are operational and reporting.

⛳️ Requirements

• Extensive experience with OpenTelemetry and observability backends.

• Practical expertise with SLOs, error budgets, actionable alerting, and incident management.

• Strong proficiency with Kubernetes and infrastructure as code (Terraform / Helm).

• Experience managing software in environments that are not fully under your control.

• Ability to produce production-quality code and conduct reviews, along with a readiness to operate what you create.

• Nice to have: Experience in deploying software to customer-hosted Kubernetes (e.g., Helm, outbound-only connectivity).

• Nice to have: Familiarity with GCP or GKE, AWS or EKS.

• Nice to have: Experience with ML multi-node/multi-cluster workloads in a production setting.

• Nice to have: Background in financial services or regulated environments.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work opportunities.

• Professional development and training programs.

• Collaborative and innovative work environment.