Principal Software Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the architecture and progression of HubSpot’s primary observability platform encompassing distributed tracing, metrics, and structured logging.

• Implement instrumentation standards throughout a diverse engineering organization.

• Spearhead the technical strategy for tracing and comprehending AI agents and ML-driven systems in a production environment.

• Define telemetry primitives, standards, and debugging workflows tailored for AI and ML systems.

• Design high-cardinality, high-throughput telemetry pipelines and storage solutions.

• Balance factors such as sampling, fidelity, retention, cost, query latency, and developer usability.

• Deliver production code and oversee design reviews and significant initiatives from prototype to full deployment.

• Create instrumentation APIs and libraries while promoting the adoption of OpenTelemetry.

• Develop tools that convert raw telemetry into actionable operational insights.

• Define SLO/SLI, alerting, and proactive incident-response methodologies.

• Collaborate with infrastructure, platform, and product engineering teams to fill signal gaps.

• Shape technical strategy and integrate observability aspects into product and operational decisions.

• Mentor senior engineers and technical leads.

• Document insights gained from major incidents and extensive migrations.

⛳️ Requirements

• Demonstrated experience in developing observability or telemetry tools for internal engineering teams.

• Experience in architecting developer platforms that support thousands of engineers within a large organization.

• Extensive knowledge of telemetry pipeline design, including high-cardinality data, dynamic sampling, query latency, retention economics, and data fidelity.

• Strong technical proficiency with OpenTelemetry, distributed tracing engines, metric backends, and large-scale log ingestion systems.

• Proven success in connecting telemetry signals to automated operational processes.

• Experience in designing real-time telemetry triggers for automated remediation, dynamic runbooks, or AI-assisted root-cause analysis across microservices environments.

• Familiarity with large-scale distributed systems, developer platform architecture, and AI observability.

• Capability to deliver production code and manage significant initiatives from start to finish.

• Experience in designing instrumentation APIs and libraries while advancing OpenTelemetry adoption.

• Knowledge in defining SLO/SLI design and alerting strategies.

• Ability to collaborate with infrastructure, platform, and product engineering teams.

• Proven ability to mentor senior engineers and tech leads.

🏝️ Benefits

• Participation in an equity plan with restricted stock units (RSUs) for eligible positions.

• Annual bonus targets under HubSpot’s bonus plan for qualifying roles.

• On-target commission for employees in eligible roles.

• Overtime compensation for qualifying positions.

• Flexible work arrangements available.

• In-person onboarding at a regional HubSpot office for members of the Engineering team.

• In-person Product Group Summit and other events for wider Product team members.

• Support for accommodations related to travel limitations or other reasons.

• Disability accommodations and assistance offered.