Senior Data Engineer – Azure

This is a fully remote position, open to applicants in Poland.

📋 Description

• Create reusable, metadata-driven data pipelines utilizing the Databricks Lakehouse architecture and PySpark.

• Design and execute comprehensive data models.

• Develop robust ETL/ELT solutions featuring advanced capabilities such as Merge operations and SCD Type 2 implementations.

• Implement incremental data loads using idempotency patterns and optimize overlap joins.

• Automate and enhance data platform processes with an emphasis on performance and reliability.

• Establish integrations with data sources and consumers employing event-driven patterns.

• Collaborate with the infrastructure engineering team to configure cloud resources.

• Initiate and execute enhancements to the data platform architecture.

⛳️ Requirements

• Expertise in Databricks: proficient in the Databricks Lakehouse architecture and PySpark development.

• Mastery of data modeling: extensive experience in defining data models that include fact tables, dimension tables, measures, grain analysis, SCD Type 2, surrogate keys, handling late-arriving records, overlap joins, and incremental loads with idempotency.

• Programming expertise: advanced skills in Python and PySpark for ETL/ELT development.

• Databricks optimization: in-depth knowledge of optimize, zOrder, Liquid clustering, ACID transactions, and performance tuning.

• Experience in event-based architecture: proven track record in designing and implementing event-driven data solutions.

• Familiarity with Azure data platform: experience with Azure-based datasets and data pipelines.

• Proficient in SQL: strong SQL capabilities for complex data transformations.

• Large-scale data processing experience: adept at handling large and complex datasets efficiently.

• CI/CD experience: background in developing automated deployment pipelines.

• Understanding of networking fundamentals: knowledge of basic networking concepts.

• Familiarity with Agile methodology: acquainted with Scrum and agile development practices.

🏝️ Benefits

• Flexible employment with remote work options.

• Opportunities to work on international projects with leading global clients.

• Possibility of international business trips.

• A non-corporate work environment.

• Language classes available.

• Access to internal and external training.

• Private healthcare and insurance coverage.

• Multisport card provided.

• Initiatives focused on well-being.