Data Engineer – Databricks

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design, develop, and oversee scalable data solutions and data pipelines utilizing Databricks.

• Collaborate with cross-functional teams to guarantee that data is reliable, accessible, and efficient for analytics and business intelligence purposes.

• Implement Medallion Architecture incorporating Bronze, Silver, and Gold lakehouse layers.

• Create frameworks for data quality featuring automated testing and monitoring.

• Utilize Delta Lake capabilities such as time travel, vacuum operations, and Z-ordering.

• Build and sustain complex ETL/ELT pipelines that process large-scale datasets on a daily basis.

• Design and execute CI/CD workflows for data pipelines leveraging Databricks Asset Bundles or similar tools.

• Develop solutions for real-time and batch data processing using Structured Streaming and Delta Live Tables.

• Enhance Spark jobs for cost efficiency and performance through cluster auto-scaling and resource management.

• Create custom integrations with Databricks APIs and external systems.

• Architect scalable data solutions utilizing Unity Catalog, Delta Lake, and Apache Spark.

• Establish data mesh architectures that incorporate governance and lineage tracking.

• Contribute to a customer-facing banking application for a prominent financial group in Japan.

⛳️ Requirements

• Over 6 years of experience in data engineering.

• Proven experience in designing and deploying production-grade data pipelines and ETL/ELT workflows.

• Proficient in Python, PySpark, and SQL.

• Extensive practical experience with AWS Glue, Lambda, Redshift, and S3.

• Strong background in data governance and management tools, including Unity Catalog and AWS Sagemaker Unified Studio.

• Demonstrated experience in data migration projects and modernizing legacy systems.

• Proficient in English communication skills.

• Databricks Certified Professional Data Engineer or a comparable certification is an advantage.

• Experience with containerization and orchestration tools such as Docker and Kubernetes is a plus.

🏝️ Benefits

• A rich and collaborative work environment.

• Opportunities for professional growth.

• Chance to contribute to an international banking app project.

• Option for remote work.