AI Product Engineer

This is a fully remote position, open to applicants in United Kingdom, +4 more countries.

📋 Description

• Develop agents that investigate incidents, identify anomalies, determine the reasons behind production issues, and utilize ClickStack as their foundation.

• Create a reusable library of skills that encompass debugging, root-cause analysis, ClickHouse querying, and workflows for incident response.

• Manage the agent stack comprehensively, covering context engineering, tool design, evaluations, tracing, and cost management.

• Construct MCP servers, SDKs, and integrations that empower customer agents to access telemetry, take actions, and maintain observability.

• Collaborate with open-source contributors and customers, troubleshoot their issues, and incorporate insights back into the product.

• Tackle challenges related to latency, costs, context-window limitations, evaluation coverage, and hallucinations in real telemetry.

• Influence the culture of a rapidly growing startup.

⛳️ Requirements

• Over 5 years of experience in software engineering.

• 1–2 years of experience working with LLM-powered systems or agents in a production environment.

• Proficient backend skills in TypeScript/Node.js and/or Python; comfortable working with both.

• Practical experience in building and deploying agents with multi-step tool utilization, planning, memory management, and error recovery.

• Skilled in designing capabilities using Markdown-based workflow encodings, Anthropic-style methodologies, or comparable techniques.

• Familiarity with MCP, including server construction, tool design, and managing authentication, scoping, and observability.

• Strong evaluation practices, including golden sets, LLM-as-judge, and regression detection methodologies.

• Proficient in SQL, with the ability to write ClickHouse queries directly.

• Comfortable using Docker and Kubernetes.

• Active engagement in open source and the developer community.

• Bonus: experience with production agents in observability, incident response, or Site Reliability Engineering (SRE).

• Bonus: experience in agent observability, including tracing, cost attribution, evaluation pipelines, or OpenTelemetry.

• Bonus: knowledge in prompt caching, context compaction, or similar techniques for production agents.

• Bonus: familiarity with columnar databases and event ingestion pipelines.

• Bonus: contributions to or maintenance of an open-source AI/agent project.

• Bonus: knowledge of Go, Rust, or other system languages for integrations and high-throughput infrastructure.

🏝️ Benefits

• Employer contributions towards healthcare.

• Stock options available for every new team member.

• Flexible time off in the United States.

• Generous time-off policies in other countries.

• USD$500 allowance for home office setup for remote employees.

• Opportunities for engagement with colleagues during company-wide offsites.

• Flexible work environment.

• Remote-friendly work arrangements.