Staff Software Engineer – OpenTelemetry, Python instrumentation

This is a fully remote position, open to applicants in United Kingdom, +4 more countries.

📋 Description

• Develop and manage the OpenTelemetry Python SDK and its associated instrumentation libraries.

• Collaborate with OpenTelemetry maintainers, special interest groups (SIGs), users, and other vendors on specifications, APIs, and implementation choices.

• Create Python instrumentation that connects AI agent operations, model invocations, and tool executions across asynchronous code.

• Contribute to OpenTelemetry conventions tailored for generative AI, taking into account sensitive data and metric cardinality.

• Enhance the performance of the Python SDK through profiling and benchmarking.

• Assess trade-offs related to runtime overhead, compatibility, and maintainability.

• Educate others and advocate for OpenTelemetry, semantic conventions, and best practices in technology/frameworks.

• Support the technical direction of the team and Grafana Labs’ OpenTelemetry strategy.

• Take ownership of projects from undefined problems to design, implementation, testing, documentation, and release.

• Collaborate with Go-to-Market (GTM), Product, and Support teams.

• Engage in customer discussions, address instrumentation issues, and translate findings into fixes and practical guidance.

• Contribute to open-source communities and Grafana Labs' products.

⛳️ Requirements

• Over 8 years of experience in software development.

• Strong expertise in Python programming.

• Proficient in debugging production software, including language internals, libraries, and asynchronous code.

• Interest in AI applications and agent systems.

• Experience in participating in open-source communities.

• Familiarity with observability tools such as Grafana, Prometheus, and OpenTelemetry.

• Background in cloud-based software or systems engineering.

• Understanding of production environments with the ability to evaluate operational risks.

• High degree of agency and effective communication skills.

• Experience in open-source observability projects, particularly with OpenTelemetry or Prometheus (bonus).

• Knowledge of Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation (bonus).

• Experience in practical technical leadership, establishing technical direction, and guiding project teams (bonus).

• Familiarity with instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI (bonus).

• Experience in contributing to or maintaining open-source projects, including successful pull requests and community collaboration (bonus).

• Rust experience is a plus.

🏝️ Benefits

• Restricted Stock Units (RSUs).

• Fully remote work with a global remote culture.

• Opportunities for career advancement.

• Open communication and regular updates for the entire company.

• Autonomy and support in delivering exceptional work.

• Empowered and high-trust teams.

• Accessible and transparent leadership.

• In-person onboarding experience.

• 30 days of annual leave each year.

• 3 annual leave days designated for Grafana Shutdown Days.