Staff Software Engineer – OpenTelemetry, Python instrumentation

This is a fully remote position, open to applicants in Germany, +4 more countries.

📋 Description

• Develop and sustain the OpenTelemetry Python SDK along with its instrumentation libraries.

• Collaborate with OpenTelemetry maintainers, vendors, and the open-source community on specifications, APIs, and implementation choices.

• Create Python instrumentation for AI agents, model calls, tool execution, and asynchronous operations.

• Contribute to OpenTelemetry conventions related to generative AI, while taking into account sensitive data and metric cardinality.

• Enhance the performance of the Python SDK through profiling and benchmarking.

• Assess trade-offs concerning runtime overhead, compatibility, and maintainability.

• Educate others and advocate for best practices surrounding OpenTelemetry, semantic conventions, technologies, and frameworks.

• Support the technical direction and vision of the team.

• Engage in business discussions regarding OpenTelemetry strategy at Grafana Labs.

• Take ownership of projects from ambiguous problems to design, implementation, testing, documentation, and release.

• Collaborate with GTM, Product, and Support teams.

• Participate in customer discussions, investigate instrumentation challenges, and translate findings into solutions and practical recommendations.

• Mentor and coach team members.

⛳️ Requirements

• Over 8 years of experience in software development.

• Strong expertise in Python.

• Proficiency in debugging production software, language internals, libraries, and asynchronous code.

• A keen interest in AI applications and agent systems.

• Experience engaging with open-source communities.

• Familiarity with Grafana, Prometheus, and OpenTelemetry.

• Background in cloud-based software or systems engineering.

• Understanding of production environments with the capacity to evaluate operational risks.

• High agency alongside excellent communication skills.

• Experience with Rust is a plus.

• Knowledge of Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation is advantageous.

• Experience in a hands-on technical leadership role is beneficial.

• Familiarity with instrumenting AI frameworks or agent workflows is a bonus.

🏝️ Benefits

• Restricted Stock Units (RSUs).

• Fully remote work in a global culture.

• Opportunities for career advancement.

• In-person onboarding experience.

• 30 days of annual leave each year.

• 3 annual leave days set aside for Grafana Shutdown Days.

• Commitment to transparent communication and regular company-wide updates.

• Autonomy and support to deliver quality work.

• Clearly defined opportunities for personal and professional growth.