Senior AIOps Engineer

This is a fully remote position, open to applicants in Washington.

📋 Description

• Take ownership of production reliability for AI/ML services by overseeing availability and performance.

• Establish SLOs and SLIs that are linked to user experience and business impact.

• Manage recurring model refresh cycles, which include retraining, validation, and redeployment.

• Develop and execute monitoring, logging, and alerting systems for models and data pipelines.

• Integrate model and system metrics with observability tools and dashboards such as Datadog and Opik.

• Optimize AI service deployment workflows, encompassing blue/green, canary, A/B, and shadow deployments.

• Enhance CI/CD pipelines and release processes for AI services.

• Facilitate the productionization of image-based ML models and batch prediction workflows.

• Provision, deploy, configure, and maintain AI infrastructure, including AI gateways, RAG platforms, LLM observability tools, agentic workflows, and no-code agent builders.

• Create automation and tooling for model promotion, configuration, environment management, and Docker image upkeep.

• Assist with multi-business-unit infrastructure provisioning across various environments, projects, roles, and CI/CD integrations.

• Ensure operational health of Label Studio, including GCS buckets, BigQuery pipelines, databases, and SSO dependencies.

• Support bi-weekly annotation project cycles.

• Establish and manage RAG and vector database infrastructure, embedding pipelines, and context retrieval APIs.

• Optimize embedding storage and retrieval performance at scale.

• Oversee and enhance LLM token usage and costs across different providers.

• Maintain centralized cost-tracking dashboards and alerting systems across business units.

• Advise on cost-effective model routing and provider selection strategies.

• Participate in an on-call rotation for AI/ML systems.

• Triage and resolve production incidents.

• Lead and contribute to post-incident reviews.

• Develop production-grade Python services and tools on Kubernetes.

• Implement testing, code reviews, CI/CD, observability, and security best practices.

• Document standards, architecture diagrams, and operational playbooks for other teams.

• Collaborate with product managers, platform engineers, data scientists, and machine learning engineers.

⛳️ Requirements

• 4+ years of relevant experience, including a minimum of 3 years in commercial software engineering or production operations (SRE, DevOps, Platform, or ML Platform).

• Experience managing services on Kubernetes and/or major cloud platforms.

• Experience operating production systems at scale, ideally in distributed, microservices-based, or Kubernetes-based architectures.

• Strong preference for GCP experience, including GKE, BigQuery, Pub/Sub, Vertex AI, Cloud SQL, GCS, and IAM.

• Practical experience with CI/CD tools and deployment automation such as GitLab CI/CD or GitHub Actions.

• Experience with infrastructure-as-code tools, preferably Terraform.

• Proficiency with monitoring, logging, and alerting tools like Datadog, Prometheus, Grafana, or ELK/EFK.

• Familiarity with container management and Docker image lifecycle.

• Experience operating and administering self-hosted open-source tools on Kubernetes, including installation, configuration, upgrades, security hardening, backup, and disaster recovery.

• Understanding of machine learning concepts and model lifecycles.

• Familiarity with common ML tooling.

• Knowledge of LLM ecosystem tooling is a significant advantage.

• Strong foundation in computer science and programming principles.

• Ability to write high-quality software using TDD, SOLID principles, clean code, and thorough documentation.

• Experience with Python-based data and ML stacks such as NumPy, SciPy, Pandas, and Scikit-Learn is a plus.

• Experience with PyTorch is an added advantage.

• Experience in building or managing ML infrastructure is highly desirable.

• Effective communication skills with data scientists, ML engineers, and product stakeholders.

• Willingness to participate in an on-call rotation.

• Must be eligible to work in the United States; Porch Group does not consider remote applicants from Alaska, Delaware, Hawaii, Mississippi, Nebraska, Montana, New Hampshire, West Virginia, or the District of Columbia.

🏝️ Benefits

• Long-term incentive awards, subject to program guidelines and approvals.

• Comprehensive health, life, and financial wellbeing coverage.

• Three medical plan options.

• Two dental plan options.

• Vision plan.

• Voluntary Critical Illness, Hospital Indemnity, and Accident plans.

• Partially employer-funded Health Savings Account.

• Flexible Savings Accounts for healthcare, dependent care, and transportation.

• Company-paid Basic Life and AD&D insurance.

• Short- and Long-Term Disability benefits.

• Voluntary Life and AD&D plans.

• Traditional and Roth 401(k) plans with discretionary employer match.

• Employer-paid Supportlinc wellbeing program.

• Guided meditation and mindfulness exercises.

• Mental health coaching and clinical care.

• Confidential resources including will preparation.

• LifeBalance discounts on gym memberships, travel, appliances, movies, pet insurance, and more.

• Flexible paid vacation.

• Company-paid holidays, typically nine per year.

• Paid sick time.

• Paid parental leave.

• Identity theft program.

• Travel assistance.

• Fitness and other discount programs.

• Remote work arrangement.