Data Engineer, Azure Databricks

📋 Description

• Design, develop, and maintain data pipelines utilizing Azure Databricks.

• Construct and enhance data transformations using PySpark and SQL within Databricks.

• Implement and uphold Lakehouse architectures leveraging Delta Lake.

• Create ETL/ELT pipelines orchestrated via Azure Data Factory.

• Integrate data from various sources into the data platform and analytical layers.

• Design and sustain data models and data warehouse structures for analytical purposes.

• Ensure data quality, scalability, and performance for large-scale data processing pipelines.

• Collaborate with BI teams to assist Power BI and reporting platforms.

• Support and enhance existing SQL Server data platforms and ETL solutions (SSIS) when necessary.

• Contribute to the design of contemporary cloud-based data architectures.

⛳️ Requirements

• Over 3 years of experience in Data Engineering or Data Warehouse development.

• Strong proficiency with Azure Databricks.

• Experience in developing data pipelines using PySpark and Spark SQL.

• Solid understanding of distributed data processing and big data concepts.

• Experience with Delta Lake and Lakehouse architectures.

• Strong SQL capabilities and experience with SQL Server relational databases.

• Experience in constructing data pipelines using Azure Data Factory.

• Experience managing large datasets and optimizing performance.

• Nice to have:

• Knowledge of Spark optimization techniques (partitioning, caching, cluster tuning).

• Experience with structured streaming in Databricks.

• Familiarity with CI/CD pipelines for data platforms (Azure DevOps).

• Experience with Power BI.

• Experience in transitioning from traditional ETL processes to cloud architectures.

🏝️ Benefits

• Culture of Relentless Performance: join an unstoppable technology development team with a 99% project success rate and more than 30% year-over-year revenue growth.

• Competitive Pay and Benefits: enjoy a comprehensive compensation and benefits package, including health insurance and a relocation program.

• Work From Anywhere Culture: take advantage of the flexibility that remote work offers.

• Growth Mindset: benefit from various professional development opportunities, including certification programs, mentorship and talent investment initiatives, internal mobility, and internship opportunities.

• Global Impact: work on impactful projects for top global clients and help shape the future of industries.

• Welcoming Multicultural Environment: be part of a dynamic, global team and thrive in an inclusive and supportive work environment with open communication and regular team-building social events.

• Social Sustainability Values: engage with our sustainable business practices focused on five pillars: IT education, community empowerment, fair operating practices, environmental sustainability, and gender equality.