Senior Data Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Create and develop scalable cloud-native data platforms from the ground up to production.

• Design and implement near-real-time ingestion pipelines utilizing event-driven patterns.

• Establish and maintain standards for data platforms, including principles of Data Lake/Lakehouse, medallion architecture, and data contracts.

• Refactor, enhance, and modernize Spark and PySpark scripts.

• Promote best practices for code quality, testing, and CI/CD across data pipelines.

• Lead the integration of AI tools and agentic workflows within the data engineering team.

• Ensure the quality, observability, and reliability of data across pipelines and platforms.

• Design and provide self-service tools and microservices that simplify platform complexity.

• Work closely with the Customer’s scientific and engineering teams.

• Monitor the performance, reliability, and costs associated with cloud infrastructure.

⛳️ Requirements

• Over 5 years of experience in data engineering.

• More than 3 years of hands-on production experience with Databricks and Delta Lake.

• Strong knowledge of medallion architecture and the design of event-driven asynchronous pipelines.

• Advanced skills in Python, PySpark, and SQL.

• Extensive practical experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.

• Solid AWS expertise in services such as S3, IAM, KMS, AWS Batch, ECR, EventBridge, Lambda, or API Gateway.

• Familiarity with infrastructure as code and CI/CD practices using Terraform and Databricks Asset Bundles.

• Hands-on experience with Git workflows, pull request reviews, and automated testing.

• Experience in configuring secure CI pipelines using OIDC or service principals.

• Practical knowledge of MLflow, including experiment tracking, model registration, and reproducibility practices.

• Strong understanding of data quality, lineage, provenance, schema contracts, and versioning.

• Familiarity with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.

• Experience in monitoring, observability, and optimizing cloud costs.

• Upper-Intermediate English proficiency or above.

• Strong analytical skills.

• Demonstrated ownership and accountability.

• Ability to clearly convey complex technical concepts.

• Proactive problem-solving mindset.

• Focus on scalability and maintainability.

• Team-oriented and collaborative attitude.

🏝️ Benefits

• Option for remote work.

• Opportunity to engage in innovative global projects.

• Chance to contribute to groundbreaking fusion research and clean-energy sustainability.

• Exposure to engineering excellence, automation, and AI integration within the data platform ecosystem.