Principal DevOps Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Architect and enhance the Kubernetes-native platform that supports NBC broadcast production environments.

• Design platform infrastructure that streamlines provisioning, lifecycle management, and cloud infrastructure delivery at an enterprise scale.

• Model broadcast infrastructure as custom resources utilizing Crossplane compositions and custom Go functions.

• Spearhead provisioning automation across multi-account AWS environments and on-premises control rooms.

• Design, construct, and maintain Kubernetes operators, controllers, and internal platform APIs using Go.

• Develop custom Crossplane providers that integrate with enterprise platforms such as NRCS, Venafi, and Infoblox.

• Oversee resource lifecycles and approval workflows.

• Lead cloud networking, DNS strategies, cross-account connectivity, VPC topology, and dynamic network routing.

• Collaborate with broadcast systems engineers, system integrators, and external vendors.

• Automate bare-metal compute configurations with Puppet and incorporate proprietary vendor solutions.

• Author RFCs, guide architectural decisions, mentor engineers, and establish CI/CD and testing strategies.

• Manage authorization models, hierarchical RBAC, resource identifiers, and identity integrations.

• Propel GitOps continuous delivery utilizing Flux, Kustomize, and Helm.

• Oversee configuration-as-code for compute fleets through Puppet.

• Design observability and alerting stacks.

• Supervise remote desktop/VDI connectivity, authentication, credential management, and gateway routing.

• Contribute upstream to open-source projects and enhance cloud-native solutions.

⛳️ Requirements

• Over 10 years of experience in designing, building, and operating production infrastructure and cloud-native platforms at an enterprise scale.

• Strong expertise in Go, including systems-level programming and API servers.

• Extensive experience in creating Kubernetes controllers/operators using controller-runtime and kubebuilder.

• In-depth knowledge of Kubernetes, including CRD/XRD generation, operators, informers, admission webhooks, and RBAC.

• Significant production experience with Crossplane, covering composite resources, composition functions, and custom Crossplane providers in Go.

• Extensive production experience with AWS multi-account architectures, cross-account networking, and identity federation.

• Proficiency with EKS, EC2, VPC, IAM, STS, SSM, Secrets Manager, Route 53, and S3.

• Production experience with GitOps tooling, specifically Flux or ArgoCD.

• Practical experience with Puppet, including module development, PuppetDB, Hiera, and r10k.

• Experience in designing REST APIs with middleware patterns and OAuth/JWT authentication.

• Understanding of information security, IAM trust chains, least-privilege policies, JWT lifecycles, and secrets abstraction.

• Experience with designing observability platforms using Grafana, Prometheus/Mimir, Loki, OpenTelemetry, Alloy, or Prometheus Node Exporter.

• Working knowledge of PostgreSQL, SQLite, or similar relational databases, including schema design, migrations, and query optimization.

• Ability to present architectural decisions, engage with vendors, and produce technical documentation.

• Familiarity with broadcast/media production workflows and live production constraints is preferred.

• Experience with Crossplane function SDK and Kubernetes disaster recovery is preferred.

• Familiarity with VDI solutions, machine identity workflows, and PKI certificate management is preferred.

• Experience with hybrid DNS, software-defined networking, Envoy Gateway, or Gateway API is preferred.

• Familiarity with k6, KUTTL, SOPS, Air, kind, or colima is preferred.

• Ability to script in Bash and PowerShell.

• Active contributions to open-source projects, especially in the CNCF/Kubernetes ecosystem, are preferred.

🏝️ Benefits

• Medical insurance

• Dental insurance

• Vision insurance

• 401(k)

• Paid leave

• Tuition reimbursement

• Discounts and perks

• Bonus eligibility