Senior Backend Engineer – Databases

This is a fully remote position, open to applicants in Canada, +1 more country.

📋 Description

• Lead the technical strategy on the most challenging problems within the Tempo roadmap.

• Execute substantial technical projects that span multiple months, from problem identification to implementation.

• Manage components related to data ingestion, storage, querying, and metrics generation.

• Create and present design documents, and engage in design reviews.

• Evaluate trade-offs involving performance, cost, and complexity, and maintain documentation of decisions made.

• Develop structured, deterministic, and discoverable APIs for users, agents, products, and integrators.

• Take responsibility for operational outcomes against SLOs, such as P99 write latency, incident recurrence, and total cost of ownership per GB ingested.

• Enhance automation, parameterized rollouts, actionable alerts, and Zero Ops practices.

• Collaborate with teams including Product, App Observability, Asserts, Drilldown, and Grafana Assistant.

• Mentor and guide junior and mid-level engineers through code reviews, design feedback, and collaborative work.

• Participate in on-call duties, incident response, and post-incident analysis.

• Engage with the open-source community, review contributions from external sources, and contribute to the Tempo project.

• Work on improving Tempo's performance, autoscaling, query ranges, trace aggregation, customer limits, and multi-cell operations.

⛳️ Requirements

• Proven experience in delivering intricate, multi-month projects covering design, delivery, and operational aspects.

• Extensive hands-on experience in developing and managing distributed data systems in a production environment.

• Familiarity with ingestion pipelines, storage engines, query execution, or related systems.

• Strong commitment to software craftsmanship, ensuring clean, reliable, and high-performance software development.

• Proficient in Go, or a willingness to learn; experience with Rust, C, or C++ is relevant.

• Previous experience managing production services and being part of an on-call rotation.

• Ability to minimize operational inefficiencies and regard SLOs as key product features.

• Excellent communication skills in a fully remote, asynchronous setting.

• Experience with tracing, OpenTelemetry, or large-scale observability systems is a plus.

• Background in designing query languages, SQL/TraceQL-like engines, or APIs intended for programmatic use is advantageous.

• Familiarity with columnar storage formats like Parquet or analytical on-disk formats designed for specific purposes is beneficial.

• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes is a plus.

• Experience in building solutions for AI/LLM consumers is advantageous.

• Contributions to or maintainership in open-source projects is a plus.

• Familiarity with Grafana, Prometheus, Loki, or Tempo in an on-call context is an advantage.

• Experience working within a fully remote, globally distributed team is a plus.

🏝️ Benefits

• Restricted Stock Units (RSUs).

• Company-funded budget for the use of modern AI coding assistants.

• Access to state-of-the-art models from OpenAI, Anthropic, and Google.

• 100% remote work culture with a global reach.

• Opportunities for career advancement.

• In-person onboarding experience.

• 30 days of annual leave each year.

• 3 days of annual leave specifically designated for Grafana Shutdown Days.