Senior Datadog Architect – Implementation Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and implement a comprehensive Datadog AWS integration across Development, UAT, and Production environments.

• Execute integrations utilizing AWS CloudFormation StackSets, AWS Organizations, and secure cross-account IAM roles.

• Develop and sustain the CloudWatch Logs-to-Datadog pipeline using Datadog Forwarder Lambda along with CloudWatch subscription filters.

• Configure the polling of Datadog AWS metrics across various AWS accounts.

• Activate CloudWatch Transaction Search and set up the Datadog AWS X-Ray integration.

• Guarantee that AgentCore trace and span data are accessible and searchable within Datadog.

• Implement distributed tracing and telemetry utilizing AWS X-Ray and OpenTelemetry.

• Create dashboards driven by tags, categorized by environment, AWS account, application, and AI solution.

• Construct dashboards to monitor AI token consumption, platform usage, performance, reliability, and cost.

• Define, configure, and fine-tune Datadog monitors and alert thresholds.

• Establish notification routing, escalation pathways, and procedures for production support.

• Diagnose issues related to missing logs, metrics, traces, permissions, tags, and cross-account integrations.

• Work in collaboration with security, cloud infrastructure, platform, and application teams on IAM and compliance assessments.

• Provide production support while maintaining the implemented Datadog platform.

• Deliver functional integrations, operational log-forwarding and metrics-polling pipelines, searchable AgentCore traces and spans, live dashboards, optimized alerts, notification routing, and support documentation.

⛳️ Requirements

• Extensive hands-on experience in implementing Datadog within multi-account AWS environments.

• Proven track record in configuring Datadog AWS metrics polling.

• Practical experience in deploying and supporting the Datadog Forwarder Lambda.

• Experience with configuring CloudWatch Logs subscription filters and centralized log forwarding.

• Strong understanding of AWS Organizations and organization-wide cloud deployments.

• Practical experience with CloudFormation and CloudFormation StackSets.

• In-depth knowledge of cross-account IAM roles, trust policies, permissions, and security measures.

• Experience with AWS X-Ray, OpenTelemetry, distributed tracing, metrics, logs, and spans.

• Hands-on experience in supporting AWS Lambda, Amazon ECS/Fargate, CloudWatch Logs, and CloudWatch Metrics.

• Familiarity with creating production Datadog dashboards, monitors, alerts, and notification workflows.

• Capability to implement solutions based on an established enterprise architecture.

• Experience navigating security, compliance, and IAM approval processes.

• Strong skills in troubleshooting, cross-team collaboration, and production support.

• Excellent written and verbal communication skills.

• Preferred: experience monitoring AI or generative AI solutions developed with Amazon Bedrock or Bedrock AgentCore.

• Preferred: experience in creating dashboards for token usage and cost monitoring for AI workloads.

• Preferred: familiarity with Datadog tagging strategies and AWS tag taxonomies.

• Preferred: possession of Datadog or AWS professional certifications.

• Preferred: experience in delivering observability solutions within regulated enterprise environments.

• Candidates must have personally designed and implemented Datadog across multiple AWS accounts; experience limited to dashboard-only or monitor management is not considered a strong match.

🏝️ Benefits

• Text-message hiring updates.

• Production support and support documentation as part of the engagement.