Senior Consultant – AI and Data, AWS Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Design, enhance, and implement scalable ETL/ELT pipelines utilizing PySpark, SQL, and AWS services.

• Solid comprehension of medallion architecture using the Databricks lakehouse.

• In-depth knowledge of managing the Databricks end-to-end platform (Unity Catalog, Users/Groups, security, etc.), Databricks integrations (AWS/Azure/GCP), and DevOps practices such as CI/CD pipelines (for notebooks, MLflow, etc.).

• Create and oversee data lakehouse solutions that leverage AWS S3, Glue, Iceberg, and other AWS-native components.

• Transition on-premises ETL processes to advanced AWS-based architectures focusing on performance, reliability, and cost-effectiveness.

• Implement metadata-driven ingestion frameworks and medallion/layered architecture (Bronze/Silver/Gold).

• Engage with orchestration frameworks like Astronomer (Airflow), AWS Step Functions, or AWS-managed workflows.

• Adhere to data warehouse (DW) principles and best practices for modeling and data integration.

• Collaborate with data analysts, data scientists, and BI engineers to ensure smooth data delivery.

• Conduct code reviews, troubleshoot problems, and maintain end-to-end data quality.

⛳️ Requirements

• Over 8 years of overall IT experience, with a minimum of 2 years in AWS-based data engineering.

• Extensive hands-on experience with Databricks Lakehouse Architecture.

• Proficiency in ETL pipeline development using PySpark, Scala, SQL, and Python.

• Familiarity with AWS services such as S3, Glue, Lambda, Step Functions, and CloudWatch.

• Experience in Unix/Linux environments and Iceberg table format.

• Knowledge of Astronomer (Airflow) or similar orchestration tools.

• Experience with structured and semi-structured data formats, including Parquet, JSON, CSV, and XML.

• Understanding of data warehousing concepts, including star/snowflake schemas and dimensional modeling.

• Proficient in CI/CD, version control systems (GitHub, Azure DevOps, Jenkins).

• Strong analytical, troubleshooting, and problem-solving abilities.

• Capacity to work independently while collaborating with stakeholders to deliver high-quality solutions.

🏝️ Benefits

• Opportunities to engage in diverse, significant, and industry-leading projects.

• Access to coaching, learning programs, and a tailored growth and development plan.

• Exposure to a collaborative, interdisciplinary work culture.

• Flexibility to manage your work in a manner that best suits you.

• Supportive colleagues and a global environment fostering continuous knowledge exchange.