Senior Data Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Create and develop scalable cloud-native data platforms from inception to production.

• Design and execute near-real-time data ingestion pipelines utilizing event-driven architectures.

• Establish and uphold standards for the data platform, including principles of Data Lake/Lakehouse, medallion architecture, and data contracts.

• Refactor, enhance, and modernize Spark and PySpark scripts for improved performance and maintainability.

• Implement best practices for code quality, testing, and CI/CD within data pipelines.

• Promote the use of AI tools and agentic workflows within the data engineering team.

• Ensure data quality, observability, and reliability throughout pipelines and platforms.

• Create and provide self-service tools and microservices that simplify platform complexity.

• Collaborate closely with the Customer’s scientific and engineering teams.

• Oversee platform performance, reliability, and costs associated with cloud infrastructure.

⛳️ Requirements

• Over 5 years of experience in data engineering.

• More than 3 years of production experience with Databricks and Delta Lake.

• Strong knowledge in medallion architecture and design of event-driven asynchronous pipelines.

• Advanced skills in Python, PySpark, and SQL.

• Extensive hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization techniques.

• Solid expertise in AWS services such as S3, IAM, KMS, AWS Batch, ECR, EventBridge, Lambda, or API Gateway.

• Experience with infrastructure as code and CI/CD practices using Terraform and Databricks Asset Bundles.

• Familiarity with Git workflows, conducting pull request reviews, and implementing automated testing.

• Experience in configuring secure CI pipelines using OIDC or service principals.

• Practical experience with MLflow, including experiment tracking, model registration, and ensuring reproducibility.

• Strong understanding of data quality, lineage, provenance, schema contracts, and version control.

• Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.

• Proficient in monitoring, observability, and optimizing cloud costs.

• Upper-intermediate level of English or above.

• Strong analytical capabilities.

• Demonstrated ownership and accountability.

• Ability to articulate complex technical concepts clearly.

• Proactive in problem-solving.

• Focused on scalability and maintainability.

• Team-oriented and collaborative mindset.

• Plus: Experience in packaging workloads using Docker containers and executing batch jobs on AWS.

• Plus: Knowledge of scikit-optimize, BoTorch, Ax, and scipy.

• Plus: Experience in developing feature sets and creating train/test splits that prevent data leakage.

• Plus: Background in regulated or high-security environments, including data residency requirements.

• Plus: Experience in disabling or restricting AI functionalities in secured environments.

🏝️ Benefits

• Employees have the flexibility to work remotely.

• Opportunity to engage in innovative global projects.

• Chance to contribute to groundbreaking research in clean energy and fusion science.

• Focus on driving engineering excellence, automation, and AI integration within the data platform ecosystem.