Data Engineer – Databricks, PySpark

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design and sustain data pipelines for the processing of large data volumes in both batch and near real-time settings.

• Utilize Databricks, PySpark, and SQL to transform, validate, and enhance data.

• Assist in the implementation of Data Lakehouse solutions adhering to engineering standards and best practices.

• Construct, test, and automate ETL processes and data workflows.

• Enhance data quality, organization, and reliability across various platforms.

• Collaborate with engineers, analysts, and stakeholders to provide scalable solutions.

• Diagnose, optimize, and oversee data pipelines.

• Keep abreast of contemporary data engineering technologies.

• Ensure all deliverables comply with quality standards and project specifications.

⛳️ Requirements

• Minimum of 2 years of experience in Big Data development and database design.

• Proficient hands-on experience with SQL.

• Strong skills in Python development.

• Practical experience with PySpark.

• Experience in developing and maintaining data pipelines within Databricks.

• Familiarity with Delta Lake.

• Knowledge of Medallion Architecture (Bronze, Silver, Gold layers).

• Strong analytical and problem-solving abilities.

• Intermediate or higher proficiency in English.

• Familiarity with AWS services, including S3, is advantageous.

🏝️ Benefits

• Remote work options available.

• Opportunity to expand expertise in Databricks, PySpark, Delta Lake, and scalable data platforms.

• Collaboration with seasoned data engineering professionals.

• Involvement in enterprise-scale solutions for a global IT consulting firm and a leading AdTech/data analytics client.