Senior Monitoring and Observability Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and oversee telemetry pipelines from start to finish utilizing Dynatrace, Datadog, and Splunk.

• Implement instrumentation for applications and infrastructure to capture metrics, logs, and traces based on OpenTelemetry standards.

• Create observability solutions for microservices, containers, and cloud-native workloads within AWS/GovCloud environments.

• Construct dashboards and enable golden-signal monitoring with correlation among metrics, events, logs, and traces.

• Supervise observability platforms and address issues related to collectors, agents, and pipelines.

• Establish and refine production alerts with defined ownership, severity levels, and links to runbooks.

• Participate in on-call rotations and engage in incident response bridges, employing telemetry-driven triage and root-cause analysis.

• Integrate alert routing, deduplication, and maintenance-window management with ServiceNow and paging tools.

• Take part in post-incident reviews and report on trends related to MTTD/MTTA.

• Deploy and configure observability agents and collectors utilizing Ansible, Ansible Tower, and AAP.

• Develop GitLab CI/CD pipelines for observability-as-code and transition existing Jenkins jobs to GitLab.

• Collaborate with platform engineering to instrument infrastructure provisioned via Terraform.

• Enhance versioned, peer-reviewed configuration-as-code for pipelines, dashboards, and alerts.

• Assist in managing tagging, lifecycle governance, retention policies, telemetry costs, ingest volumes, cardinality, data masking, auditing, and continuous monitoring.

• Maintain architecture documentation and runbooks while working within a SAFe framework using ServiceNow, Jira, and Confluence.

⛳️ Requirements

• U.S. Citizenship is required.

• Must be able to obtain and maintain the necessary Public Trust level clearance.

• Bachelor's Degree with 8 years of relevant experience, a Master's Degree with 6 years, or a High School diploma (or equivalent) with 12 years of experience.

• At least 4 years of hands-on experience with enterprise observability, monitoring, and alerting platforms.

• Practical experience with Dynatrace, Datadog, and/or Splunk, including tasks related to instrumentation, dashboard creation, and alert design.

• Proficiency in metrics, logs, and traces, along with experience in OpenTelemetry-based instrumentation.

• Experience working within AWS and/or AWS GovCloud.

• Familiarity with containerized and cloud-native workloads.

• Experience with Ansible / Ansible Tower / AAP for automated deployment of agents and configurations.

• Knowledge of GitLab CI/CD and Jenkins for developing observability-as-code pipelines.

• Strong scripting and automation skills in Python, Bash, or Go.

🏝️ Benefits

• Possible eligibility for overtime compensation.

• Possible eligibility for shift differentials.

• Possible eligibility for a discretionary bonus.