Senior Observability Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and oversee a cohesive observability framework encompassing metrics, logs, traces, and events.

• Spearhead First-of-a-Kind (FOAK) implementations, transforming new observability technologies into secure, repeatable, and production-ready patterns.

• Establish enterprise standards for telemetry pipelines, data retention policies, high-cardinality controls, and cost management of observability.

• Define and manage Service Level Indicators (SLIs), Objectives (SLOs), and error budgets.

• Act as the senior technical escalation point, lead major P1/P2 incident war rooms, and perform evidence-based Root Cause Analysis (RCA).

• Minimize Mean Time to Detect (MTTD) and Mean Time to Recovery (MTTR) while reducing alert noise through event correlation, dynamic thresholds, and dependency mapping.

• Promote Observability-as-Code and drive infrastructure automation initiatives.

• Automate the deployment, configuration, and self-healing workflows for monitoring agents and telemetry collectors.

• Integrate observability platforms with IT Service Management (ITSM) tools like ServiceNow, as well as Netcool and CI/CD pipelines.

• Design observability frameworks for Docker, Kubernetes, microservices, and multi-cloud environments including Azure, AWS, and GCP.

• Correlate application APM telemetry with Kubernetes control planes, pods, nodes, and infrastructure dependencies.

• Ensure telemetry pipelines are secure-by-design, implementing RBAC, TLS, secrets management, and image scanning.

• Lead Knowledge Transfer (KT) programs, manage vendor transitions, and facilitate operational readiness handovers for global 24x7 teams.

• Mentor cross-functional engineering teams and influence the technology roadmaps of the enterprise.

⛳️ Requirements

• Over 12 years of comprehensive IT experience, including a minimum of 5 to 7 years in roles such as Lead Architect, SRE, or Principal Observability Engineer within a large enterprise environment.

• Demonstrated success in transitioning organizations from traditional monitoring to proactive, automated observability platforms.

• Practical expertise in developing scalable and secure telemetry pipelines along with time-series databases.

• Significant experience in leading FOAK rollouts and managing intricate vendor/operations transition (KT) programs.

• Proficient in IBM Instana, Grafana (Enterprise & Alloy), Prometheus, OpenTelemetry, Telegraf, and InfluxDB.

• Familiarity with SolarWinds, Netcool, Elastic/Splunk.

• Knowledge of Kubernetes, Docker, OpenShift, and cloud platforms such as AWS/Azure/GCP.

• Experience with Linux (RHEL), Windows Server, VMware, Citrix VDI, load balancers, and edge proxies.

• Skills in Ansible, Terraform, Python, Bash, Webhooks, and CI/CD tools (GitHub Actions/GitLab/Jenkins).

• Understanding of ServiceNow, ITIL 4, and advanced Major Incident Management practices.

• Preferred certifications include CKA (Certified Kubernetes Administrator), Cloud Architect (AWS/Azure), or specific APM/Observability vendor certifications.

🏝️ Benefits

• CLT hiring with a 40-hour weekly workload.

• Remote work opportunities.

• Health Insurance through SulAmérica Prestige (available for legal dependents).

• Dental Insurance via SulAmérica (available for legal dependents).

• Life Insurance from Prudential (equivalent to 24 times salary).

• Private Pension plan with Metlife (up to 6% company match).

• Meal Voucher and Internet allowance through Flash (R$1.000,00 per month).

• Employee Assistance Program available.

• Wellness program offered.

• Individual performance compensation program based on eligibility.

• Equity grant under the Associate Equity Appreciation Program, subject to eligibility.