Senior Software Engineer, Observability Platform

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and develop an observability platform that encompasses metrics, logs, distributed traces, and events across various regions and environments.

• Take ownership of components from telemetry collection to storage, querying, and presentation.

• Manage observability as an internal product, including published interfaces, versioned client libraries, deprecation paths, and platform SLOs.

• Create and sustain OpenTelemetry instrumentation libraries, collector deployments, semantic conventions, context propagation, and sampling strategies.

• Engineer high-volume collection, enrichment, redaction, routing, reliability, backpressure management, and tiered-retention pipelines.

• Integrate observability tools with CI/CD, service catalog, incident management, ticketing, chat, and feature-flag systems.

• Design event-driven and agent-assisted self-healing remediation workflows, incorporating human-in-the-loop approval gates.

• Implement SLOs, error budgets, golden-signal alerting as code, and strategies for reducing alert noise.

• Oversee the golden path for service instrumentation, including scaffolding, templates, Terraform modules, and CI checks.

• Monitor platform adoption metrics, onboarding duration, dashboard time-to-value, query performance, and service cost.

• Develop documentation, reference architectures, worked examples, onboarding paths, workshops, office hours, and game days.

• Lead code reviews and architecture discussions, establish instrumentation standards, mentor engineers, and cultivate instrumentation champions.

• Utilize AI tools for coding, testing, design, troubleshooting, and observability in AI and agentic systems.

• Participate in the on-call rotation, conduct root-cause analysis, and convert incidents into lasting improvements.

• Report to the Team Lead of Observability Engineering and collaborate with the Principal Engineer on the direction of the telemetry platform.

⛳️ Requirements

• 5+ years of experience in building and operating highly scalable and available distributed systems, platform services, or observability infrastructure.

• 5+ years of programming experience in Go, Python, Java/Kotlin, or TypeScript/Node.js.

• Experience in constructing internal platforms, developer tooling, or an internal developer platform like Backstage.

• Hands-on expertise in metrics, logs, and distributed tracing on a significant commercial or open-source observability platform.

• Solid understanding of cardinality, sampling, cost mechanics, SLOs, and production alerting.

• Practical experience with OpenTelemetry, including collectors, both automatic and manual instrumentation, semantic conventions, and context propagation.

• Experience in building high-volume data or telemetry pipelines.

• Strong skills in REST API and integration engineering, including OAuth, authentication patterns, webhooks, and event-driven architectures.

• Advanced expertise in AWS and Kubernetes, including EKS, ECS, EC2, Lambda, and managed messaging/eventing services.

• Familiarity with Terraform and GitOps-based workflows.

• Proven track record of technical teaching through workshops, onboarding curricula, internal courses, conference or meetup presentations, or enhancing team capabilities.

• Experience in measuring and driving the adoption of platforms or tools.

• At least 6 months of professional experience applying AI to boost engineering productivity.

• History of autonomously leading large projects from ambiguous problems to production.

• Degree in Computer Science, Engineering, or equivalent practical experience.

• Legal eligibility to work in the U.S. on an ongoing basis.

🏝️ Benefits

• Employer-subsidized medical, vision, and dental coverage for full-time employees.

• 401k Match: 50% of your contribution up to the first 6% of eligible pay.

• Monthly stipend to enhance work and productivity.

• Flexible Time Away Program, along with Sick Time Off.

• Smartsheet-sponsored life insurance, short-term disability, and long-term disability plans.

• 12 paid holidays annually.

• Up to 24 weeks of Parental Leave.

• Personal paid Volunteer Day.

• Opportunities for professional growth and development, including Udemy online courses.

• Company-funded perks, including a counseling membership, local retail discounts, and a personal Smartsheet account.

• Teleworking options from any registered location in the U.S. (role-specific).

• Market-competitive incentive opportunities.