Principal SRE

This is a fully remote position, open to applicants in United States.

📋 Description

• Take responsibility for the availability and durability of essential services within production environments and ensure accountability for outcomes.

• Lead projects from the identification of issues through to resolution.

• Provide insights on service health metrics utilizing SLIs, SLOs, and error budgets; collaborate with service owners to define and refine SLIs/SLOs and create error budget policies and alerting protocols.

• Engage in leadership on-call rotations, incident management, and post-incident evaluations.

• Adhere to ITIL/OSS methodologies for incident, change, problem, and capacity management.

• Create automation for routine operational tasks to minimize manual involvement and reduce toil.

• Work alongside engineering teams on storage design and performance benchmarks in relation to product SLAs.

• Establish monitoring, logging, and alerting frameworks using tools such as Prometheus, Grafana, Catchpoint, and ELK; integrate runbooks with FireHydrant.

• Collaborate with CI/CD pipelines, configuration management, and infrastructure as code tools, including Terraform, Ansible, and Jenkins.

• Develop Bash, Python, or Go scripts to enhance system reliability and efficiency.

• Partner with engineering, customer success, product, and operations teams to ensure resilient system design and operations.

• Engage with customers to align stability priorities with their requirements and product experiences.

• Collaborate with vendors and service providers to resolve service issues and monitor SLA performance.

• Document systems, share insights, and foster a culture of reliability-minded engineering.

• Lead cross-functional teams to identify gaps in observability and operational processes and establish prioritization strategies.

• Recognize recurring issues, suggest long-term enhancements, and formulate action plans.

• Advocate for reliability-focused practices within development and operations teams.

⛳️ Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).

• Over 10 years of experience in site reliability, systems engineering, or operations; experience with hyperscalers is a plus.

• Exposure to large-scale, production-grade systems, including the establishment of operational procedures or runbooks and mentoring others on these practices.

• Strong Linux systems administration and troubleshooting expertise.

• Demonstrated ability to utilize AI tools to enhance platform stability and scalability.

• Understanding of storage systems, performance, sharding, and storage algorithms.

• Experience with load balancing services, including IPVS.

• Security-focused operations (patching, hardening, identifying vulnerabilities).

• Familiarity with service reliability concepts — monitoring, alerting, incident response, and root cause analysis.

• Proficient in at least one scripting language (Python, Bash, or Go).

• Experience with containers (Kubernetes, Docker) and microservices concepts is essential; comfortable operating in a Kubernetes/Vitess-on-K8s environment (kubectl, mysqlsh, Vitess keyspaces).

• Familiarity with Hashicorp products (Nomad, Vault, Terraform).

• Preference for experience in a SaaS, service provider, or distributed systems environment.

• Awareness of ITIL/OSS practices and SLO/SLAs is preferred.

• Strong problem-solving abilities and a willingness to learn new technologies are advantageous.

• Experience with cloud platforms (AWS, GCP, or Azure) is preferred.

🏝️ Benefits

• Comprehensive healthcare coverage for families, including dental and vision.

• Competitive salary and 401K plan.

• RSU grants available for full-time employees.

• Employee Stock Purchase Plan (ESPP) program.

• Flexible vacation policy.

• Maternity and paternity leave.

• MacBook Pro provided for work, along with a generous stipend for personalizing your workstation.

• Childcare bonus (for human children only).

• Support for fertility treatments.

• Learning and development programs.

• Commuter benefits.

• A work culture that promotes a healthy work-life balance.