Senior Data Engineer – Python, PySpark, Kafka

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Design and implement the lakehouse infrastructure that supports virtual physical therapy platforms.

• Lead the transition to the Apache Iceberg format.

• Build resilient data pipelines and develop infrastructure suitable for AI applications.

• Work collaboratively with Backend Engineering teams on API integrations and the establishment of formal data contracts.

• Offer extensive support to the Core Team in leveraging new capabilities of the data platform.

• Ensure the creation of clean, governed, and accessible data pipelines that fuel machine learning models and AI-driven products throughout the platform.

⛳️ Requirements

• Proven expertise in Python and PySpark for large-scale data processing.

• Practical experience with data lake formats such as Iceberg, Delta Lake, or Hudi.

• Strong understanding of Kafka and event-driven architectures.

• Established experience in constructing and orchestrating data pipelines at scale.

• Advanced SQL skills with a solid foundation in data modeling.

• Familiarity with workflow orchestration tools like Airflow, Dagster, or similar.

• Platform-centric mindset: focused on creating solutions for widespread organizational application rather than individual use.

• Ownership mentality: dedicated to resolving issues thoroughly.

• Excellent communication skills: capable of conveying complex technical ideas to non-technical audiences.

• Highly collaborative: thrives in teamwork with backend engineers, data engineers, and analysts.

• Pragmatic approach: effectively balances ideal solutions with realistic delivery timelines.

• Experience in building and sustaining AI-ready data infrastructure.

• Ability to utilize AI coding assistants and LLMs to enhance development speed.

• Proficiency in English at a minimum of B2 Upper Intermediate level.

🏝️ Benefits

• Flexible work arrangements.

• Opportunities for professional development.