Data Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design, develop, and sustain scalable and dependable data platforms and pipelines for analytical and operational applications.

• Construct distributed data processing workflows utilizing Apache Spark, pySpark, and Databricks.

• Create and enhance cloud-native data solutions by leveraging Azure and AWS services.

• Model and refine data structures to support analytics, reporting, and downstream utilization.

• Integrate data from various heterogeneous sources while ensuring quality, consistency, and traceability.

• Implement controls for data quality, schema validation, lineage tracking, and monitoring processes.

• Enhance the performance, scalability, reliability, and cost-effectiveness of data pipelines and infrastructure.

• Collaborate closely with scientists, analysts, and engineering teams to translate business and research needs into technical solutions.

• Contribute to CI/CD processes, infrastructure automation, and adhere to engineering best practices.

• Investigate and introduce new technologies and methods within the data engineering landscape.

⛳️ Requirements

• 3+ years of professional experience in Data Engineering.

• 2+ years of practical production experience with Databricks.

• Strong expertise in Delta Lake, including medallion architecture and table optimization.

• Advanced proficiency in Python, pySpark, and SQL.

• In-depth practical knowledge of Databricks components such as Unity Catalog, Lakeflow Jobs / Workflows, and job optimization.

• Hands-on experience with AWS services including S3, IAM, KMS, and familiarity with at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway.

• Experience in designing event-driven and asynchronous data pipelines.

• Strong understanding of Infrastructure as Code and CI/CD methodologies.

• Practical experience with Terraform and Databricks Asset Bundles.

• Familiarity with Git workflows, pull request reviews, and automated testing.

• Understanding of secure authentication methods such as OIDC and service principals.

• Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices.

• Strong grasp of data quality, lineage, schema contracts, and provenance concepts.

• Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQX.

• Experience in monitoring and optimizing data platform reliability and cloud expenditures.

• Upper-Intermediate or higher proficiency in English.

🏝️ Benefits

• Employees have the option to work remotely.

• Continuous learning opportunities.

• Chance to contribute to groundbreaking scientific initiatives.

• Work with cutting-edge technologies and highly skilled international teams.