Senior Software Engineer II – LLM Observability

This is a fully remote position, open to applicants in Canada.

📋 Description

• Take charge of the complete development of AI observability functionalities, covering everything from data ingestion to in-product visualizations.

• Design and enhance the data model and infrastructure for monitoring agents and their ecosystems throughout the Honeycomb product.

• Create systems that analyze and highlight AI-specific signals, such as costs, token usage, errors, and anomalies.

• Partner with product and design teams to develop user-friendly, intuitive features that enhance significant user workflows.

• Provide guidance on best practices for investigating and comprehending AI system behaviors across various conversations and workloads.

• Engage in rapid feedback cycles and experimentation, sharing both achievements and setbacks.

• Collaborate within a small, cross-functional engineering team that includes a dedicated product manager and designer.

• Deliver features that customers actively utilize in production.

⛳️ Requirements

• Extensive software engineering experience with a proven history of shipping and owning features from start to finish.

• Experience in delivering customer-facing features at a SaaS organization, rather than solely internal tools.

• Full-stack expertise in backend development using Go and/or Python.

• Proficiency in frontend development with TypeScript and React.

• Proven experience collaborating closely with product managers and designers.

• Regular, hands-on use of AI tools in everyday engineering tasks.

• Experience in on-call duties; readiness to join the on-call rotation after approximately 90 days.

• Demonstrated ability to navigate ambiguity in a fast-paced environment where requirements evolve and experimentation is essential.

• Bonus: Experience in developing LLM or agent-powered features in a production setting.

• Bonus: Practical experience with observability tools such as OpenTelemetry.

• Must confirm identity and eligibility to work.

• Honeycomb is unable to currently sponsor or support visa transfers.

🏝️ Benefits

• Generous equity with an employee-friendly stock program.

• Transparent compensation based on levels relative to experience.

• Unlimited paid time off (PTO).

• Distributed-first culture and mindset.

• Stipend for home office, co-working, and internet expenses.

• Comprehensive benefits coverage for employees, with additional options available for dependents.

• Up to 16 weeks of paid parental leave, regardless of the path to parenthood.

• Annual development allowance.

• Reasonable accommodations provided during the hiring process.