Senior Observability Systems Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the technical direction and standards for observability throughout the organization, which includes instrumentation conventions, alerting philosophy, and design patterns for dashboards.

• Provide mentorship to engineers on the Observability team and serve as a technical resource for the SRE, Storage Engineering, and Network Engineering teams.

• Lead design reviews and represent the team during cross-functional architectural discussions.

• Assess build-versus-buy options for observability and AIOps tools, taking into account data sovereignty and compliance factors.

• Create and maintain design documentation, runbooks, and records of architectural decisions.

• Oversee and refine the metrics pipeline, utilizing VictoriaMetrics, Prometheus, and Consul-based service discovery.

• Manage and enhance the OpenSearch, Logstash, and Filebeat logging framework across Backblaze data centers.

• Design and extend distributed tracing utilizing Jaeger and OpenTelemetry.

• Establish and uphold standards for Grafana dashboards.

• Promote the enhancement of internal application telemetry in collaboration with application teams.

• Define strategies for capacity and retention within the telemetry stack.

• Manage infrastructure-as-code for observability deployments across various Kubernetes clusters.

• Develop and review automated playbooks for Debian-based bare metal servers.

• Collaborate with SRE on Consul-based service discovery and automation of scrape targets.

• Direct Ceph storage observability, focusing on OSD health, performance metrics, and alert integration.

• Work alongside Network Engineering on BGP, routing, and network-level observability.

⛳️ Requirements

• Minimum of 8 years in a systems engineering, SRE, DevOps, or observability-centric role, with a proven track record of technical leadership on a platform or infrastructure team.

• Extensive hands-on experience with Prometheus, VictoriaMetrics, or comparable metrics platforms, including proficiency in PromQL, Alertmanager, and managing these systems at scale.

• Experience in designing and managing log aggregation pipelines, such as Elasticsearch/OpenSearch, Logstash, Filebeat, or similar technologies.

• Working knowledge of distributed tracing (e.g., Jaeger, OpenTelemetry, or equivalents), including strategies for instrumentation, not just operation.

• Skilled in Python and/or Go for automation and tooling purposes.

• Familiarity with Kubernetes, including workload deployment and understanding pod networking and storage.

• Experience with infrastructure-as-code tools and configuration management.

• Strong grasp of Linux internals, preferably with Debian/Ubuntu.

• Proven experience mentoring engineers and elevating the technical standards within a team.

• Exceptional written and verbal communication capabilities.

• Experience leading the assessment or implementation of AIOps tools, including alert correlation, deduplication, or automated root cause analysis.

• Background in building, deploying, or operating AI agents or agentic AI systems.

• Familiarity with LLM APIs (like Anthropic, OpenAI) or self-hosted LLM inference in an infrastructure context.

• Knowledge of Ceph storage, including OSD management, pool configuration, and CRUSH maps.

• Experience with incident management platforms.

• Understanding of BGP, network observability, or experience in multi-DC environments.

• Competence in secrets management within production environments.

• Experience operating in regulated or compliance-focused environments.

• Previous experience in a storage-oriented company or within a hyperscale infrastructure setting.

🏝️ Benefits

• Target annual bonus of 10%.

• Restricted Stock Units (RSUs).

• Commitment to Equal Opportunity Employment.

• Values of diversity, equity, and inclusion.

• Commitment to fostering a sense of belonging within the workforce.