Principal Development Engineer

This is a fully remote position, open to applicants in Idaho, +2 more states.

📋 Description

• As a Principal Software Development Engineer, you will serve as the primary technical authority across the AI Platform within HCD.

• You will take ownership of the platform decisions that every agent relies on.

• Establish the engineering standards that the entire portfolio adheres to.

• Tackle the most challenging architectural issues throughout the Agent Development Lifecycle.

• Act as a force multiplier for a team of skilled engineers who are delivering substantial work at a rapid pace.

• Manage the LLM gateway strategy from end to end, including governance of model access.

• Conduct latency benchmarking across various routing layers.

• Lead the design of the Model Context Protocol (MCP) farm for authenticated, scalable tool integrations across all agentic applications.

• Define and uphold reference architectures, template repositories, SDKs, and the AI QuickStart that pave the way for agent delivery across HCD and OGAI.

• Drive the strategy for the Skills and Plugin Marketplace.

• Ensure a platform-agnostic reusable component model that functions across different teams and technology stacks.

• Take ownership of and continuously refine the ADLC framework.

• Implement engineering quality gates across all types of agents.

• Guarantee that the evaluation framework design occurs prior to the commencement of development.

• Manage structured rollback procedures, canary deployment patterns, and production readiness standards that prevent instability from affecting the entire portfolio.

• Provide principal-level technical leadership across the active agent portfolio.

• Own the most complex cross-cutting technical challenges within the portfolio.

• Facilitate delivery across multiple concurrent work streams by addressing architectural dependencies.

• Lead the deployment automation strategy to ensure agents transition smoothly from development to production via a standardized, auditable pipeline.

• Own the observability mandate, promoting adoption across all production agents and establishing evaluation standards that reflect actual business outcomes.

• Build and maintain CI and CD pipelines that integrate LLM evaluations.

• Define and enforce the AI Software Development Life Cycle (SDLC).

• Oversee the AI spend tracking system, monitoring costs per team, per agent, and per model, with automated reporting for senior leadership.

• Manage the AI metrics dashboard, showcasing adoption trends, deployment frequency, token consumption, and per-agent business KPIs to executive stakeholders.

• Provide leadership in compliance and security design for agents handling sensitive data.

⛳️ Requirements

• 9+ years of professional experience in software engineering.

• In-depth expertise in Python, with strong cloud infrastructure experience, preferably in Azure, and practical knowledge of API gateway patterns and LLM routing.

• Proven ability to influence architectural and engineering standards across various teams and work streams without direct management authority.

• 3+ years of experience in a principal, staff, or equivalent senior individual contributor role is preferred.

• Practical experience with AI observability tools such as Arize, LangSmith, or Phoenix, along with the capability to design evaluation pipelines that assess what truly impacts business outcomes.

• Demonstrated success in designing and operating production AI systems — including LLM APIs, agent frameworks, RAG architectures, or RPA automation — at a significant scale.

• Strong understanding of security and compliance requirements for AI systems that manage sensitive data, including PHI and PII in a regulated environment.

• A bachelor's degree is preferred, along with specialized training or relevant professional qualifications.

🏝️ Benefits

• Medical, dental, and vision coverage.

• Paid time off.

• Retirement savings options.

• Wellness programs.

• A comprehensive benefits package designed to support the physical, emotional, and financial well-being of colleagues and their families.