Senior Data Engineer

This is a fully remote position, open to applicants in Ireland.

📋 Description

• Create and implement data pipelines utilizing Python (PySpark), ensuring both performance and reliability.

• Ingest and merge various data sources (both real-time and batch) into the data lake, emphasizing data quality.

• Maintain and enhance data models along with the broader data estate codebase.

• Supervise daily operations and monitoring of cloud data platforms, primarily Azure / Microsoft Fabric.

• Diagnose and resolve issues within data processes and pipelines.

• Collaborate with engineers, analysts, and business stakeholders to aid data initiatives and perform ad hoc analyses.

• Contribute to ongoing improvement through learning and development efforts.

• Execute additional tasks as needed to meet business requirements.

⛳️ Requirements

• Bachelor's degree in IT, Computer Science, or a related discipline.

• A minimum of 5 years’ experience as a Data Engineer, specializing in designing and building scalable ETL processes for both batch and real-time data lakes/warehouses.

• Practical experience with cloud data services; familiarity with Microsoft Synapse and Fabric is a plus.

• Strong SQL proficiency with experience in relational databases and complex querying.

• Skilled in using compute tools such as Databricks or Jupyter.

• Experience in orchestrating pipelines using ADF, Fabric Pipeline, Airflow, AWS Step Functions, or similar tools.

• Coding proficiency in PySpark, Python, Scala, or other equivalent programming languages.

• Knowledge of CI/CD tools (Git, Azure DevOps, Jenkins, etc.) is essential.

• Exposure to distributed systems (Kafka, Spark, Hive, Hadoop) and BI tools (e.g., Power BI) is advantageous.

• Ability to excel in a fast-paced, agile environment.

• Strong analytical and problem-solving capabilities.

• Excellent communication and collaboration skills.

• Self-driven with a commitment to continuous learning.

🏝️ Benefits

• Clear scope with no ambiguity regarding deliverables.

• Opportunity for repeat engagements based on performance.