Data Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design, develop, and maintain scalable and dependable data platforms and pipelines for both analytics and operational applications.

• Create distributed data processing workflows utilizing Apache Spark, pySpark, and Databricks.

• Develop and enhance cloud-native data solutions by leveraging Azure and AWS services.

• Model and optimize data structures for analytics, reporting, and downstream usage.

• Integrate data from various heterogeneous sources while ensuring data quality, consistency, and traceability.

• Implement data quality controls, schema validation, lineage tracking, and monitoring procedures.

• Enhance the performance, scalability, reliability, and cost-effectiveness of data pipelines and infrastructure.

• Collaborate closely with scientists, analysts, and engineering teams to translate business and research needs into technical solutions.

• Contribute to CI/CD processes, infrastructure automation, and engineering best practices.

• Investigate and introduce new technologies and methodologies within the data engineering domain.

⛳️ Requirements

• Over 3 years of professional experience in Data Engineering.

• More than 2 years of hands-on production experience with Databricks.

• Strong knowledge of Delta Lake, including medallion architecture and table optimization techniques.

• Advanced skills in Python, pySpark, and SQL.

• In-depth practical knowledge of Databricks components such as Unity Catalog, Lakeflow Jobs / Workflows, and job optimization strategies.

• Experience with AWS services, including S3, IAM, KMS, and familiarity with at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway.

• Background in designing event-driven and asynchronous data pipelines.

• Solid understanding of Infrastructure as Code and CI/CD methodologies.

• Practical experience with Terraform and Databricks Asset Bundles.

• Experience with Git workflows, pull request reviews, and automated testing processes.

• Familiarity with secure authentication methods such as OIDC and service principals.

• Practical knowledge of MLflow, experiment tracking, model registry, and reproducibility practices.

• Strong grasp of data quality, lineage, schema contracts, and provenance concepts.

• Familiarity with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQX.

• Experience in monitoring and optimizing data platform reliability and managing cloud expenses.

• Upper-intermediate or higher proficiency in English.

• Plus: Experience with packaging workloads using Docker and executing batch workloads in AWS.

• Plus: Familiarity with ECR and containerized data processing solutions.

• Plus: Experience with experiment optimization libraries like scikit-optimize, BoTorch, Ax, or scipy.

• Plus: Understanding of ML feature engineering and techniques to prevent leakage.

• Plus: Experience in regulated or high-security environments.

• Plus: Knowledge of data governance, compliance, and data residency requirements.

• Plus: Experience in disabling or restricting AI-assisted tools/features in secure settings.

🏝️ Benefits

• Option for remote work.

• Opportunities for continuous learning.

• Chance to contribute to groundbreaking scientific initiatives.

• Work with modern technologies and highly skilled global teams.