Software Engineer L5 – AI Observability, Agent Evaluation

This is a fully remote position, open to applicants in Washington.

📋 Description

• Develop an observability framework and platform capabilities that deliver metrics, logs, and distributed traces for online inference, batch scoring, feature pipelines, and agent orchestration.

• Create tools that empower teams to independently monitor model performance, data quality, drift, and degradation within their systems.

• Design and enhance evaluation frameworks for LLM and agentic systems, focusing on response quality, grounding and hallucination, task success, tool usage, trajectory accuracy, LLM-as-a-judge, and human-in-the-loop scoring.

• Lead assessments of build versus buy options for observability and evaluation tools.

• Manage SDKs, connectors, and APIs that integrate vendor platforms into a unified interface.

• Develop reusable libraries, SDKs, and templates that ensure observability and evaluation become standard for new systems.

• Provide dashboarding, alerting, and SLO/SLI foundational elements and ready-to-use templates.

• Collaborate with engineering, product, machine learning, and data teams to transform requirements into reusable platform capabilities.

• Enable ML/AI practitioners to oversee model quality in production, assess LLM and agentic systems, and implement vendor tools via a self-service platform.

⛳️ Requirements

• Background in software, AI/ML, or platform engineering, with practical experience in production observability, monitoring, or ML/LLM evaluation.

• Demonstrated experience in creating standards, libraries, SDKs, or frameworks that are adopted by other teams.

• Proficient in coding with Python and at least one of Java, Go, or Scala, with experience in developing production services.

• Practical experience in operating ML models in production (both online serving and batch), including managing drift, model performance, and data quality.

• Experience with modern observability stacks such as Prometheus/Grafana, Datadog, OpenTelemetry, ELK/OpenSearch, Jaeger/Tempo, or similar technologies.

• Strong understanding of distributed systems, microservices, and familiarity with at least one leading cloud platform (AWS, GCP, or Azure).

• Ability to collaborate across teams including ML, data, infrastructure, and product, while effectively communicating about system behavior, quality, and risk.

• Experience with vendor integration and VPC deployment.

• Possess an AI-Native Engineering Mindset, utilizing AI tools as integral to their workflow to expedite design, development, testing, and code review.

• Nice to have: Practical experience with ML/LLM observability and evaluation tools such as Arize, Braintrust, LangFuse, Weights & Biases, Galileo, Vertex AI Model Monitoring, or SageMaker Model Monitor.

• Nice to have: Experience in constructing or delivering LLM/GenAI applications and assessing them, which includes prompt/result logging, evaluation metrics, LLM-as-a-judge, and human-in-the-loop review.

• Nice to have: Experience in evaluating agentic systems, focusing on tool usage, multi-step reasoning, and measuring trajectory/task success.

🏝️ Benefits

• Health Plans

• Mental Health support

• 401(k) Retirement Plan with employer match

• Stock Option Program

• Disability Programs

• Health Savings and Flexible Spending Accounts

• Family-forming benefits

• Life and Serious Injury Benefits

• Paid leave of absence programs

• Full-time hourly employees accumulate 35 days annually for paid time off, applicable for vacations, holidays, and sick leave.

• Full-time salaried employees receive immediate access to flexible time off.