Principal Architect – Cloud and Observability

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Take charge of the enterprise observability reference architecture, encompassing metrics, logs, traces, and events across all environments (both cloud and on-premises).

• Lead the OpenTelemetry-first instrumentation approach, including standard libraries, semantic conventions, collector topologies (DaemonSet, gateway, sidecar), and pipeline architecture.

• Construct and manage telemetry pipelines utilizing Grafana Mimir, Loki, and Tempo, featuring multi-tenant configurations, retention policies, and capacity planning.

• Establish metrics for reliability measurement: SLOs, SLIs, error budgets, and alerting frameworks consistently across all business units.

• Oversee the integration of observability tools with incident management systems (ServiceNow ITOM, xMatters).

• Champion telemetry schema standards to ensure teams produce data that is not only technically compliant but also valuable downstream.

• Develop and sustain reference architectures for our hybrid infrastructure: OpenShift on-prem with KVM/libvirt and Dell PowerFlex storage, alongside Azure, AWS, and GCP.

• Lead standardization efforts around workload identity and federation using SPIFFE/SPIRE, along with cloud-native IAM patterns to transition away from static secrets.

• Offer insights on selecting compute runtimes -- containers vs. VMs vs. bare metal vs. serverless -- providing a clear decision-making framework for teams.

• Advance FinOps maturity by integrating cost data into the observability framework, establishing unit economics, and moving towards open billing standards like FOCUS.

• Identify opportunities where AI/ML can provide practical benefits within our observability stack, including anomaly detection, root cause analysis, log clustering, and intelligent alerting.

• Set observability standards for AI-driven systems (agents, RAG pipelines), addressing latency, token costs, model drift, and associated signals.

• Ensure that new AI-driven platforms are equipped with correct instrumentation from day one.

• Engage in cross-functional architecture working groups that focus on observability and hybrid cloud standards.

• Publish architecture decision records and reference implementations that are practical for teams to utilize.

• Mentor architects and platform engineers; conduct architecture reviews to elevate standards across the organization.

• Collaborate with security and compliance teams on HIPAA, SOX, and PCI requirements as they pertain to telemetry and cloud infrastructure.

• Represent CVS Health during vendor evaluations and maintain connections with the open-source community (CNCF, OpenTelemetry, Grafana Labs).

⛳️ Requirements

• 10+ years of experience in infrastructure, cloud architecture, platform engineering, or SRE.

• 8+ years of architectural experience in observability, cloud infrastructure, or both within a large enterprise.

• Strong expertise with at least two of Azure, AWS, or GCP, covering networking, identity, compute, and storage.

• 5+ years of hands-on experience with Kubernetes in production environments (OpenShift, EKS, AKS, or GKE).

• 5+ years working with OpenTelemetry or similar frameworks (collectors, SDKs, semantic conventions, pipeline architecture).

• 5+ years of experience with observability platforms: Grafana/Mimir/Loki/Tempo, Prometheus, Datadog, Splunk, Dynatrace, or equivalent tools.

• Experience in defining SLOs/SLIs and developing alerting strategies at an organizational scale.

• Proven ability to write architecture standards that have been adopted and adhered to by other teams.

• Excellent communication skills, capable of engaging effectively with both engineers and senior leadership.

🏝️ Benefits

• Medical, dental, and vision coverage.

• Paid time off.

• Retirement savings options.

• Wellness programs.

• Comprehensive benefits package tailored to support the physical, emotional, and financial well-being of colleagues and their families.