Lead Observability Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Serve as the primary contact for customer engineering leadership while leading an embedded TechPod as Pod Leader.

• Conduct a two-month Observability Maturity Assessment.

• Evaluate the observability landscape across various vendors, agents, collectors, query surfaces, data volumes, and operating models.

• Confirm metric cardinality, active series, scrape target health, log volumes, trace sampling, retention, and telemetry fidelity.

• Develop total cost of ownership comparisons for current and future states.

• Design an AWS-native observability target architecture utilizing OpenTelemetry and AWS services.

• Collaborate with Security, Legal, and Engineering teams on telemetry classification and policy-driven retention.

• Map existing capabilities to AWS-native equivalents and pinpoint gaps.

• Define and execute performance-parity tests for query performance, alert latency, and data fidelity.

• Plan and oversee migration execution, which includes pipeline cutover, porting log transformations to OpenTelemetry, rebuilding dashboards, and deduplicating/rebuilding alerts.

• Establish OpenTelemetry conventions, collector topology, and sampling strategies.

• Revise service ownership tagging for telemetry cost attribution.

• Analyze platform expenditures, licensing, and commitment structures while informing the vendor renewal strategy.

• Maintain a working cadence with customer observability leadership.

• Generate assessment reports, target architecture, cost models, migration plans, executive summaries, and present findings to engineering leadership.

⛳️ Requirements

• Minimum of 8 years in SRE, DevOps, Observability, or Platform Engineering.

• At least 4 years of experience managing production observability platforms.

• Previous experience in a technical lead, staff, or principal-level role.

• Extensive experience in running metrics, logging, and tracing platforms at a large scale.

• Advanced production experience with Prometheus and Thanos, Cortex, or Mimir.

• Proficiency in PromQL, recording rules, remote write, and cardinality management.

• Practical experience with Datadog or a similar commercial platform.

• Production experience with CloudWatch, CloudWatch Logs, Container Insights, X-Ray or Application Signals, Amazon Managed Service for Prometheus, and Amazon Managed Grafana.

• Experience with OpenTelemetry Collector or ADOT, including pipeline design, processors, tail and head sampling, and multi-backend export.

• Background in designing and operating log pipelines using Vector, Fluent Bit, Logstash, or Firehose.

• Strong production experience with EKS or Kubernetes.

• High proficiency with Terraform.

• Experience in designing SLO-based alerting and integrating with incident management tools like PagerDuty.

• Ability to create defensible TCO models from usage data and pricing.

• Strong scripting skills in Python, Go, or Bash.

• Proven ability to assess unfamiliar environments and deliver recommendations rapidly.

• Capability to articulate technical, cost, and compliance tradeoffs to both technical and executive stakeholders.

• Experience with vendor migrations, dashboards and alerts as code, Grafana ecosystem tools, continuous profiling, data governance, analytics platforms, consumer-scale platforms, or consulting is a plus.

• Certifications such as Prometheus Certified Associate, OpenTelemetry Certified Associate, AWS Certified DevOps Engineer – Professional, AWS Certified Solutions Architect – Professional, or CKA are highly valued.

• EverOps offers remote hiring in the United States.

• Must possess legal authorization to work for any employer in the U.S.

🏝️ Benefits

• Fully remote workplace.

• Unlimited paid time off.

• Equity: Become a true owner of the company.

• 401K with company contribution.

• Sponsored healthcare.

• Access to training and certification programs.