Middle Data Engineer, Azure Databricks

This is a fully remote position, open to applicants in Cyprus.

📋 Description

• Design, develop, and maintain data pipelines utilizing Azure Databricks.

• Construct and enhance data transformations using PySpark and SQL within Databricks.

• Implement and uphold Lakehouse architectures with Delta Lake.

• Create and sustain ETL/ELT pipelines managed through Azure Data Factory.

• Integrate data from various sources into the data platform and analytical layers.

• Design and manage data models and data warehouse structures for analytics purposes.

• Ensure data quality, scalability, and performance of large-scale data processing pipelines.

• Collaborate with BI teams to support Power BI and reporting platforms.

• Assist in evolving existing SQL Server data platforms and SSIS ETL solutions as necessary.

• Contribute to the design of contemporary cloud-based data architectures.

⛳️ Requirements

• Over 3 years of experience in Data Engineering or Data Warehouse development.

• Proficiency with Azure Databricks.

• Experience in developing data pipelines using PySpark and Spark SQL.

• Strong understanding of distributed data processing and big data concepts.

• Familiarity with Delta Lake and Lakehouse architectures.

• Excellent SQL skills along with experience in SQL Server relational databases.

• Experience in building data pipelines with Azure Data Factory.

• Ability to handle large datasets and optimize performance.

• Nice to have: Techniques for Spark optimization, including partitioning, caching, and cluster tuning.

• Nice to have: Experience with structured streaming in Databricks.

• Nice to have: Knowledge of CI/CD pipelines for data platforms using Azure DevOps.

• Nice to have: Familiarity with Power BI.

• Nice to have: Experience in migrating traditional ETL processes to cloud architectures.

• Strong analytical and problem-solving capabilities.

• Ability to thrive in collaborative environments and adapt as needed.

• Commitment to ongoing learning and professional growth with a focus on cloud computing.

• Effective communication skills with both technical and non-technical stakeholders.

• Proactive in identifying enhancements and suggesting solutions.

• Comfortable working in dynamic environments where priorities and technologies are subject to change.

🏝️ Benefits

• Health insurance.

• Relocation program.

• Flexibility for remote work / Emphasis on a Work From Anywhere culture.

• Opportunities for professional development.

• Access to certification programs.

• Mentorship and talent investment initiatives.

• Internal mobility options.

• Internship opportunities.

• An inclusive and supportive work environment.

• Open lines of communication.

• Regular team-building social events.

• Commitment to sustainable business practices focusing on IT education, community empowerment, fair operating practices, environmental sustainability, and gender equality.