Data Engineer, Spark

This is a fully remote position, open to applicants in Poland.

📋 Description

• Create and sustain a robust data processing platform tailored for automotive data, focusing on scalability and reliability.

• Design and execute data pipelines capable of processing substantial data volumes in both streaming and batch formats.

• Enhance data workflows to guarantee efficient ingestion, processing, and storage utilizing technologies such as Spark, Cloudera, and Airflow.

• Utilize data lake technologies (e.g., Iceberg) to effectively manage both structured and unstructured data.

• Collaborate with cross-functional teams to comprehend data requirements and ensure smooth integration of data sources.

• Monitor and resolve issues within the platform, maintaining high availability, performance, and accuracy in data processing.

• Utilize cloud services (AWS) for managing infrastructure and scaling processing workloads.

• Produce and uphold high-quality Python (or Java/Scala) code for data processing tasks and automation.

⛳️ Requirements

• Minimum of 4 years of commercial experience in implementing, developing, or maintaining Big Data systems, along with data governance and management processes.

• Proficient programming skills in Python (or Java/Scala), focusing on clean code and OOP design.

• Practical experience with Big Data technologies such as Spark, Cloudera, Kafka, Data Platform, Airflow, NiFi, Docker, and Iceberg.

• Strong understanding of dimensional data and data modeling techniques.

• Experience in implementing and deploying solutions within cloud environments.

• Consulting experience with exemplary communication and client management skills, including prior direct client interactions as a consultant.

• Capability to work autonomously and take responsibility for project deliverables.

• Fluent in English (minimum C1 level).

• Bachelor’s degree in a technical or mathematical field.

• Preferred: Experience with MLOps frameworks like Kubeflow or MLFlow, and familiarity with Databricks and/or dbt.

🏝️ Benefits

• Work within a supportive team of enthusiastic AI & Big Data professionals.

• Engage with leading global enterprises and innovative startups on international projects.

• Enjoy flexible working arrangements, enabling you to work remotely or from modern offices and coworking spaces.

• Accelerate your career development through defined paths, knowledge-sharing initiatives, language classes, and sponsored training or conferences, including partnerships with Databricks offering industry-leading training materials and certifications.

• Choose your preferred cooperation model: B2B or a contract of mandate, with access to 20 fully paid vacation days.

• Participate in team-building activities and utilize the integration budget.

• Celebrate work anniversaries, birthdays, and significant milestones.

• Access comprehensive medical and sports packages, vision care, and well-being support services including psychotherapy and coaching.

• Receive full work equipment for optimal productivity, including a laptop and other necessary devices.

• Experience a seamless onboarding process with a dedicated buddy, beginning your journey in our friendly, supportive, and autonomous culture.