Senior Software Development Engineer

This is a fully remote position, open to applicants in California, +16 more states.

📋 Description

• Design, develop, and scale evaluation frameworks tailored for Zillow’s agentic AI experiences.

• Construct tracing, observability, and quality measurement systems for AI agents in production.

• Establish platform infrastructure that allows domain teams across Zillow to utilize the Evals platform efficiently.

• Collaborate with applied scientists and machine learning engineers to incorporate new AI evaluation features into production systems.

• Transform how Zillow assesses AI quality, reliability, trustworthiness, and product impact.

• Keep up with emerging agentic AI paradigms, evaluation strategies, and LLM tools, and translate these into platform innovations.

• Assist in scaling, ensuring reliability, optimizing performance, managing incidents, and controlling costs for the evaluation layer.

• Employ first-principles thinking to tackle ambiguous challenges and rapidly iterate on innovative solutions.

• Design, deploy, and manage scalable production ML services.

• Assess third-party AI platforms and make informed build-versus-buy decisions regarding platform infrastructure.

• Foster collaboration across applied science, ML, product, design, and engineering teams.

⛳️ Requirements

• Over 4 years of backend engineering experience in designing, deploying, and managing scalable production ML services.

• Proven experience in building platform infrastructure or developer-facing services utilized by multiple teams.

• Proficient in APIs, user experience, reliability, and serving internal customers.

• Hands-on experience with ML evaluations and observability frameworks such as Databricks MLflow, LangSmith, Braintrust, or Promptfoo.

• Skilled in evaluating third-party AI platforms and making well-reasoned build-versus-buy decisions.

• Demonstrated fluency in collaborating across applied science, ML, product, design, and engineering disciplines.

• Comfortable working with LLMs, agentic systems, and evaluation frameworks.

• Familiar with orchestration tools like LangChain, LangGraph, or MCP.

• Experience using AI-assisted development tools.

• Strong sense of ownership and a growth-oriented mindset.

• Advanced degree in Computer Science, Machine Learning, or a related field, or equivalent hands-on experience in developing products using frontier LLMs, multimodal models, or agent-based systems (preferred).

• Experience in designing and managing evaluation pipelines, including offline batch evaluations, online scoring, cost/latency/signal trade-offs, and linking user signals to training data (preferred).

🏝️ Benefits

• Equity awards based on experience, performance, and location.

• Flexible distributed work model through Zillow's Cloud HQ.

• Opportunity to work from a location of your choice.

• Inclusive, innovative, and engaging workplace culture.

• Empowerment for employees to pursue growth.

• Equal employment opportunity for all.

• Disability or special-needs accommodations available.

• Consideration for qualified applicants with arrest or conviction records in line with applicable state and local laws.