Principal Engineer – AI

This is a fully remote position, open to applicants in California, +4 more states.

📋 Description

• Develop and manage the fundamental infrastructure for the Enterprise AI Platform, which includes the AI Gateway, Policy Engine, Identity Fabric, AI Registry, Guardrails Runtime, and AI Observability.

• Take full ownership of platform functionalities from design to deployment, ensuring effective operation of production infrastructure.

• Construct, configure, fortify, scale, upgrade, and govern enterprise-level platform capabilities.

• Create a federated AI Registry featuring self-service onboarding, lifecycle workflows, and external registry integration.

• Design infrastructure for policy-as-code, including policy compilation, distribution, approval workflows, and simulation.

• Develop telemetry pipelines, trace correlation, lineage-stamped traces, and a tamper-evident audit lake to provide regulator-compliant evidence.

• Execute certification workflows, compliance scoring, decommission governance, and evidence generation processes.

• Establish gateway runtime capabilities across AWS and Azure, focusing on policy evaluation, routing, residency, budget/quota controls, circuit breaking, and cross-region failover.

• Construct multi-stage guardrails for moderation, prompt-injection defense, PII protection, output validation, hallucination detection, policy enforcement, and agentic workload safeguards.

• Implement AI workload identity, token exchange, trust boundaries, Entra Agent ID integration, identity propagation, and cross-cloud token federation.

• Enhance operability and defensibility through instrumentation, SLOs, latency budgets, failure-mode planning, and runtime evidence.

• Develop APIs, interfaces, and integrations for domains, DevOps pipelines, and enterprise systems.

• Evaluate emerging AI infrastructure, foundation-model access patterns, and standards to ensure cost-aware engineering decisions.

• Establish engineering standards, review designs and code, mentor team members, and enhance technical expertise.

• Collaborate with AI Developer Experience, AI Security, AI SDLC, and the Senior AI Architect.

• Engage in on-call operations for the services you own.

⛳️ Requirements

• A Bachelor's degree in Computer Science, Software Engineering, or a related technical field (Master's degree preferred).

• 8+ years of experience in software/platform engineering for Principal level, or 5+ years for Senior level.

• Significant experience in building and managing shared platform services at an enterprise scale.

• Proficient in operating production infrastructure with defined SLOs and on-call responsibilities.

• Experience with one or more of the following: API gateways/traffic enforcement, policy-as-code and authorization, workload identity/zero-trust, observability and telemetry pipelines, or audit/compliance data platforms.

• Strong understanding of distributed systems and platform engineering fundamentals.

• Proficient programming skills in Python and/or Go; familiarity with TypeScript/Java is a plus.

• Experience with cloud-native architecture across AWS and Azure, including containers/Kubernetes, service mesh, and Infrastructure as Code (CDK, Terraform, CloudFormation/ARM).

• Solid practices in CI/CD, GitOps, and DevSecOps.

• Familiarity with Git-based workflows (Bitbucket/GitHub), Jira, and Confluence.

• Specialized knowledge in policy/authorization, identity/zero-trust, observability/audit, gateway/guardrails, or registry/portal technologies.

• Understanding of GenAI platform patterns, such as LLM/AI gateways, RAG, agentic patterns, foundation models, embeddings, and guardrails.

• Knowledge of Bedrock, Azure OpenAI, SageMaker, MLflow, LangChain, and/or LlamaIndex.

• Background in Responsible AI, AI/data governance, privacy, cloud security, and IAM as applied to AI workloads.

• Willingness to participate in an on-call rotation for owned services.

🏝️ Benefits

• Performance-based incentives.

• Discretionary bonuses.

• Health insurance.

• Tuition reimbursement.

• Accident insurance.

• Life insurance.

• Retirement savings plans.

• Comprehensive training and coaching.

• Support from management.

• Opportunities for network building.

• Access to tools and resources to achieve new milestones.

• Reasonable accommodations for individuals with disabilities.