AWS Data Engineer, ETL Expert

This is a fully remote position, open to applicants in India.

📋 Description

• Construct and maintain AWS data pipelines.

• Create ETL/ELT workflows utilizing AWS Glue, PySpark, Python, and SQL.

• Ingest data from various sources such as Salesforce, databases, APIs, and S3.

• Load processed data into Amazon Redshift and data lake storage.

• Enhance the performance of Athena and data lake.

• Transform JSON/CSV data into Parquet format.

• Apply Snappy compression.

• Develop effective partitioning strategies.

• Oversee modern data lake architecture.

• Collaborate with Apache Iceberg tables.

• Execute migrations from traditional Parquet tables.

• Facilitate schema evolution, time travel, and ACID transactions.

• Provide production support and troubleshooting.

• Investigate Glue jobs experiencing unexpected slowdowns.

• Diagnose Lambda timeouts.

• Resolve missing records and data quality challenges.

• Address Redshift performance issues.

• Construct AWS infrastructure using Terraform.

• Develop reusable modules.

• Manage Auto Scaling Groups, ALBs, IAM, Lambda, Secrets Manager, S3, Redshift, and DynamoDB.

• Configure secret rotation using Lambda.

• Implement least-privilege access in IAM.

• Utilize enterprise backup tools such as Rubrik.

• Validate backup jobs.

• Conduct restore operations.

• Support disaster recovery testing.

• Validate source and target record counts.

• Maintain audit/control tables in DynamoDB.

• Verify checksums and identify duplicate records.

• Investigate data discrepancies reported by business users.

• Employ AWS Services: S3, Glue, Athena, Lambda, Redshift, DynamoDB, EventBridge, Step Functions, CloudWatch, Secrets Manager, IAM, Auto Scaling Groups, and Application Load Balancer.

⛳️ Requirements

• Relevant Experience: 9+ years.

• Mandatory Tech Stack: Source - Salesforce (ECRM & OSC), Backup - Grax, Cloud - AWS (EC2, S3, CloudWatch, DynamoDB, RDS, Secrets Manager, ALB, ASG), Infrastructure as Code - Terraform.

• File Format: Parquet.

• Database: Blue Database.

• Query Language: SQL.

🏝️ Benefits

• Competitive salary.

• Comprehensive health benefits.

• Opportunities for professional development.

• Flexible work hours.