Staff Observability Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Oversee comprehensive observability design, telemetry schemas, and OpenTelemetry pipeline architectures.

• Collaborate with engineering and product teams to establish SLIs, SLOs, error budgets, and requirements for production readiness.

• Design AI observability solutions for Amazon Bedrock and agentic workflows.

• Monitor model selection, latency, retries, token usage, tool execution, and cost distribution.

• Ensure traceability across user requests, prompts, model calls, retrieved context, downstream services, and final outputs.

• Uphold data privacy and security protocols.

• Create dashboards, alerts, SLOs, runbooks, and operational perspectives.

• Lead incident diagnosis, performance enhancement, and economic viability.

• Specify measurable signals for response quality, groundedness, guardrail outcomes, and task fulfillment.

• Engage in an on-call rotation approximately 20% of the time.

• Facilitate failure injection, production testing, and capacity planning.

• Work in conjunction with SRE, MLOps, data engineering, security, product teams, and non-technical stakeholders.

⛳️ Requirements

• 6+ years of experience in Site Reliability, Observability, or Platform Engineering supporting 24x7x365 production environments.

• Extensive experience with OpenTelemetry and platforms such as Datadog, Sumo Logic, Prometheus, or Grafana.

• Proven experience in designing observability for distributed systems, APIs, event-driven architectures, and asynchronous workflows.

• Practical experience in instrumenting AI/ML or GenAI systems.

• Proficient with Amazon Bedrock or comparable platforms.

• Knowledge of agentic workflows, prompt engineering, vector databases, RAG architectures, LangChain, LlamaIndex, or SageMaker.

• Proficient in Java, Python, or Go.

• Strong expertise in AWS, including cloud networking, IAM, security, and service limits.

• Experience with IaC, CI/CD, and containers, such as Terraform, Kubernetes, Argo CD, Jenkins, or GitHub Actions.

• Familiarity with data modeling, telemetry pipelines, cardinality management, retention, responsible AI, and data privacy measures.

• Excellent communication, mentoring, and cross-functional leadership abilities.

🏝️ Benefits

• Eligibility for performance-based bonuses.

• Equity options available.

• 15 hours of paid work time each year for volunteer activities and programs.

• Remote-first working environment.

• Stipends for new hires to set up home office.

• Equity grants for new hires.

• Paid time off.

• Extended paid parental leave.

• Company holidays.

• Health and wellness programs encompassing physical, mental, social, and financial well-being.

• Opportunities for professional development.

• Access to over 6,000 on-demand courses.

• Programs for manager and leadership development.

• Health and related benefits programs.