Principal Site Reliability Engineer

This is a fully remote position, open to applicants in Iowa, +4 more locations.

📋 Description

• Oversee the observability and reliability initiatives for Government Systems operating on Azure Government.

• Establish the technical roadmap for assessing and enhancing organizational reliability.

• Lead the shift towards SLO-based reliability engineering practices.

• Provide guidance to development teams on application telemetry and instrumentation strategies.

• Act as the primary technical expert on Datadog, focusing on dashboard creation, alert management, and SLO tracking.

• Collaborate with Product and Development teams to recommend service level objectives.

• Serve as a senior escalation point during incidents.

• Enhance and advance the broader Site Reliability Engineering (SRE) framework.

• Develop disaster recovery plans, including advanced fault injection scenarios.

• Offer guidance on the SRE training curriculum and related content.

• Establish and adhere to priorities set by the SRE Manager.

• Mentor and train junior SREs.

⛳️ Requirements

• U.S. citizenship is required.

• Ability to obtain and maintain a Public Trust clearance.

• Extensive hands-on experience with Datadog, including dashboards, alerting, SLO/error budget tracking, and telemetry pipelines.

• Proven track record of leading an organization’s shift to SLO-based reliability engineering.

• Experience in advising development teams on instrumentation and telemetry strategies.

• Profound technical knowledge of full-stack observability techniques.

• Capability to create templated observability dashboards and manage configuration as code.

• Proficient in setting and influencing organization-wide SLOs and Error Budgets.

• Comfortable working in both Linux and Windows server environments.

• Experience in preparing and evaluating SREs for on-call readiness.

• Familiarity with conducting psychologically safe post-mortems.

• Experience serving as a senior on-call escalation point.

• Skilled in identifying and resolving root causes of excessive alerting and pager load.

• Advanced understanding of systems design, high availability, deployment, recovery techniques, failure modes, and system behaviors.

• Specialized expertise in denial-of-service protection, containerization, or a related field.

• Ability to make resilience and recoverability decisions based on load testing data and fault injection results.

• Experience documenting SRE design practices and contributing to an SRE framework.

• Familiarity with contributing to disaster recovery plans and runbooks.

• Capable of designing and leading disaster recovery practice scenarios and incident response exercises.

• Experience planning chaos engineering scenarios across memory, network, I/O, or distributed systems.

• Ability to build consensus on engineering and architecture standards for platforms and CI/CD components.

• Experience in promoting adoption and standardization, including inner sourcing.

• Skilled in creating and sharing SRE best practices and training materials.

• Comfortable coaching fellow SREs and identifying developmental gaps.

• Ability to identify and eliminate large-scale toil.

• Experience using GitHub Copilot for troubleshooting, automation, or reliability tools is preferred.

• Interest in defining AI-assisted development best practices within an SRE context is a plus.

🏝️ Benefits

• Annual incentive bonus.

• Country-specific benefits.

• Accommodation or adjustment support during the hiring process.