Data Engineer, Azure Databricks

📋 Description

• Design, develop, and sustain data pipelines utilizing Azure Databricks.

• Construct and enhance data transformations using PySpark and SQL within Databricks.

• Implement and uphold Lakehouse architectures leveraging Delta Lake.

• Develop ETL/ELT pipelines managed through Azure Data Factory.

• Integrate data from various sources into the data platform and analytical layers.

• Create and manage data models and data warehouse structures for analytics purposes.

• Ensure the quality, scalability, and performance of extensive data processing pipelines.

• Collaborate with BI teams to facilitate Power BI and reporting platforms.

• Support and enhance existing SQL Server data platforms and ETL solutions (SSIS) as needed.

• Contribute to the design of contemporary cloud-based data architectures.

⛳️ Requirements

• Over 3 years of experience in Data Engineering or Data Warehouse development.

• Proficient experience with Azure Databricks.

• Experience in developing data pipelines using PySpark and Spark SQL.

• Strong understanding of distributed data processing and big data concepts.

• Familiarity with Delta Lake and Lakehouse architectures.

• Excellent SQL skills and experience with SQL Server relational databases.

• Proven experience in constructing data pipelines using Azure Data Factory.

• Experience in managing large datasets and optimizing performance.

• Nice to have:

• Experience with Spark optimization techniques (partitioning, caching, cluster tuning).

• Experience with structured streaming in Databricks.

• Knowledge of CI/CD pipelines for data platforms (Azure DevOps).

• Familiarity with Power BI.

• Experience in transitioning from traditional ETL processes to cloud architectures.

🏝️ Benefits

• Culture of Relentless Performance: join an unstoppable technology development team with a 99% project success rate and over 30% year-over-year revenue growth.

• Competitive Pay and Benefits: enjoy a comprehensive compensation and benefits package, including health insurance and a relocation program.

• Work From Anywhere Culture: benefit from the flexibility that comes with remote work.

• Growth Mindset: take advantage of various professional development opportunities, including certification programs, mentorship and talent investment programs, and internal mobility and internship opportunities.

• Global Impact: work on significant projects for top global clients and help shape the future of industries.

• Welcoming Multicultural Environment: be part of a vibrant, global team and thrive in an inclusive and supportive workplace with open communication and regular team-building social events.

• Social Sustainability Values: join our sustainable business practices focused on five pillars, including IT education, community empowerment, fair operating practices, environmental sustainability, and gender equality.