Staff Software Engineer – OpenTelemetry, Python instrumentation

This is a fully remote position, open to applicants in Ireland, +3 more countries.

📋 Description

• Develop and sustain the OpenTelemetry Python SDK along with its instrumentation libraries.

• Collaborate with maintainers and vendors to define specifications, APIs, and implementation strategies.

• Create Python instrumentation that links AI agent executions, model invocations, and tool operations across asynchronous code.

• Contribute to OpenTelemetry standards for generative AI while taking into account sensitive data and metric cardinality.

• Enhance the performance of the Python SDK through profiling and benchmarking.

• Assess trade-offs relating to runtime overhead, compatibility, and maintainability.

• Educate others and advocate for best practices concerning OpenTelemetry, semantic conventions, technologies, and frameworks.

• Support the technical vision and direction of the team.

• Engage in business discussions regarding OpenTelemetry strategies at Grafana Labs.

• Take ownership of projects from ambiguous challenges through design, implementation, testing, documentation, and release.

• Collaborate with GTM, Product, and Support teams.

• Participate in customer discussions, troubleshoot instrumentation issues, and transform findings into solutions and actionable guidance.

• Work with remote teams via regular video conferences.

• Contribute to Open Source communities.

⛳️ Requirements

• Over 8 years of experience in software development.

• Strong expertise in Python programming.

• Proven experience in debugging production software, language internals, libraries, and asynchronous code.

• Interest in AI applications and agent systems.

• Experience in engaging with open-source communities.

• Familiarity with observability tools such as Grafana, Prometheus, and OpenTelemetry.

• Background in cloud-based software or systems engineering.

• Understanding of production environments and the ability to evaluate operational risks.

• High level of agency and effective communication skills.

• Experience in open-source observability projects, preferably within OpenTelemetry or Prometheus (bonus).

• Familiarity with Python instrumentation methods, profiling, runtime diagnostics, or asynchronous context propagation (bonus).

• Experience in a hands-on technical leadership role (bonus).

• Experience with instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI (bonus).

• Contributions to or maintenance of Open Source projects (bonus).

• Experience with Rust (plus).

🏝️ Benefits

• Restricted Stock Units (RSUs).

• Fully remote work culture with a global reach.

• Opportunities for career advancement.

• In-person onboarding experience.

• 30 days of annual leave each year.

• 3 annual leave days specifically allocated for Grafana Shutdown Days.

• Transparent communication and frequent company-wide updates.

• Autonomy and support to produce outstanding work.

• Clearly defined paths for career growth and development.