Senior Software Engineer II – LLM Observability

This is a fully remote position, open to applicants in United States.

📋 Description

• Take full ownership of the complete development of AI observability features, encompassing everything from data ingestion to in-product visualizations.

• Design and enhance the data model and infrastructure for monitoring agents and their ecosystems throughout the Honeycomb product suite.

• Develop systems that analyze and highlight AI-specific signals, including costs, token usage, errors, and anomalies.

• Collaborate with product and design teams to craft user-friendly, intuitive features and impactful user workflows.

• Educate on best practices for exploring and comprehending AI system behaviors across various conversations and workloads.

• Engage in rapid feedback cycles and experimentation, sharing both successes and setbacks.

• Work within a small, cross-functional engineering team alongside a dedicated product manager and designer.

• Deliver features that customers actively utilize in production environments.

• Engage in on-call duties approximately 90 days after starting.

⛳️ Requirements

• Extensive software engineering experience, with a proven track record of delivering and owning features from start to finish.

• Experience in deploying customer-facing features in a SaaS environment, rather than solely internal tools.

• Full-stack expertise in backend development using Go and/or Python, as well as frontend development with TypeScript and React.

• Proven ability to work closely with product managers and designers.

• Regular, hands-on application of AI tools in daily engineering tasks.

• Experience with on-call responsibilities; will join the rotation after about 90 days.

• Demonstrated ability to navigate ambiguity in a rapidly evolving field where requirements frequently change and experimentation is essential.

• Bonus: Experience in developing LLM or agent-driven features in production.

• Bonus: Practical experience with observability tools, such as OpenTelemetry.

• Must verify identity and eligibility for employment.

• Honeycomb is not currently able to sponsor or support visa transfers.

🏝️ Benefits

• Generous equity options through an employee-friendly stock program.

• Transparent compensation structure based on levels relative to experience.

• Unlimited paid time off (PTO).

• A distributed-first culture and mindset.

• Stipend for home office, co-working, and internet expenses.

• Comprehensive benefits coverage for employees, with additional options for dependents.

• Up to 16 weeks of paid parental leave, irrespective of the path to parenthood.

• Annual budget for professional development.

• Reasonable accommodations provided during the hiring process.