AWS Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Create and develop robust, scalable data transformation pipelines utilizing SQL, DBT, and Jinja templating.

• Design and sustain data architecture and standards for Data Integration and Data Warehousing initiatives using DBT and Amazon Redshift.

• Collaborate with cross-functional teams to gather requirements and produce dimensional data models that act as a single source of truth.

• Take ownership of the entire data modeling process in DBT to empower analysts, data scientists, and BI engineers.

• Improve and maintain the analytics codebase, which includes DBT models, SQL scripts, and ERD documentation.

• Ensure the quality of data, alignment with governance standards, and operational readiness of data pipelines.

• Implement software engineering best practices such as version control, CI/CD, and code reviews.

• Optimize SQL queries for performance, scalability, and maintainability across extensive datasets.

• Apply best practices for SQL performance tuning, including partitioning, clustering, and materialized views.

• Construct and manage infrastructure as code using AWS CDK for scalable and repeatable deployments, while integrating and automating deployment workflows using AWS CodeCommit, CodePipeline, and other related DevOps tools.

• Support Agile development methodologies and collaborate with offshore teams.

• Conduct thorough data profiling on staging datasets to evaluate completeness, accuracy, consistency, timeliness, and adherence to business rules prior to downstream ingestion.

• Perform gap analysis between source, staging, and target data models to identify missing attributes, inconsistent definitions, and transformation challenges that may affect reporting and analytics.

• Collaborate with business SMEs, product owners, and analytics teams to clarify data definitions, resolve ambiguities, and prioritize the remediation of essential data gaps.

⛳️ Requirements

• Bachelor's or Master's (preferred) degree in a quantitative or technical discipline such as Statistics, Mathematics, Computer Science, Information Technology, Computer Engineering, or a related field.

• Over 4 years of experience in data engineering and analytics on contemporary data platforms.

• More than 3 years of extensive experience with DBT or similar data transformation tools, including the creation of complex and maintainable DBT models and development of DBT packages/macros.

• Strong understanding of dimensional modeling/data warehousing concepts with expertise in designing, implementing, operating, and extending enterprise dimensional models.

• Knowledge of change data capture concepts.

• Experience with AWS Services (Lambda, Step Functions, MWAA, Glue, Redshift).

• Practical experience with AWS CDK, CodeCommit, and CodePipeline for automation of infrastructure and CI/CD processes.

• Proficiency in Python or general knowledge of Jinja templating in Python and/or PySpark.

• Experience with Agile methodologies and willingness to collaborate with extended offshore teams, assisting with design and code reviews with customers.

• A collaborative team player and self-starter, with a keen attention to detail critical for this role.

🏝️ Benefits

• Work From Home