Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Europe.

📋 Description

• Design, develop, and maintain scalable data pipelines and platforms utilizing Databricks, Spark, Python, SQL, and various cloud technologies.

• Create, optimize, and support solutions for both batch and streaming data processing.

• Establish and uphold reliable, secure, and high-performance data workflows, encompassing orchestration, monitoring, and operational support.

• Construct and enhance data lake and lakehouse solutions to facilitate analytics, reporting, and machine learning projects.

• Collaborate with architects, data engineers, data scientists, analysts, and business stakeholders to convert requirements into effective technical solutions.

• Ensure the quality, integrity, availability, and performance of data across platforms and pipelines.

• Contribute to defining and implementing data engineering standards, best practices, and design patterns.

• Engage in software development lifecycle activities, including source control, testing, CI/CD, deployment, and operational support.

• Assess emerging technologies, tools, and best practices to continually enhance the data platform.

• Offer technical guidance and mentorship to junior engineers when necessary.

⛳️ Requirements

• Over 6 years of experience in Data Engineering or a relevant field.

• Extensive hands-on experience with Databricks in production settings.

• Strong background in Spark (preferably PySpark), distributed data processing, and managing large-scale data workloads.

• Advanced skills in Python and SQL.

• Proven experience in designing and constructing cloud-based data platforms and pipelines (AWS preferred).

• Experience in developing and supporting batch and streaming data processing solutions.

• Proficient in optimizing, troubleshooting, and fine-tuning data pipelines and platforms.

• Experience in building and maintaining Data Lake and/or Lakehouse architectures.

• Understanding of data governance, quality, lineage, security, privacy, and retention practices.

• Experience working in Agile environments and collaborating with cross-functional teams.

• Strong ability to work independently, take ownership of technical solutions, and provide guidance to other engineers when needed.

• Familiarity with workflow orchestration tools and modern software development practices, including Git, Docker, and CI/CD.

• Background in supporting Machine Learning, Analytics, or Data Science initiatives is a plus.

• Experience in the utilities, energy, or related sectors is a plus.

• Excellent communication, collaboration, and problem-solving abilities.

• Advanced proficiency in English (both written and spoken).

🏝️ Benefits

• Flexible work arrangements.

• Remote work options.

• Training and development programs.

• Comprehensive time-off policy.