Senior Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• The Senior Data Engineer is responsible for designing, constructing, and maintaining the data ingestion and transformation pipelines that feed into Inchcape’s central data platforms.

• This position entails the ingestion of raw data from various systems, curating the bronze and silver layers of the medallion architecture, ensuring data quality and performance, and providing dependable, reusable datasets for modeling, analysis, reporting, and business intelligence.

• Develop and sustain ETL/ELT pipelines to import data from ERP, CRM, operational, financial, and third-party sources into the Microsoft Fabric Platform - Lakehouse (OneLake) utilizing Delta Lake/Parquet.

• Establish data quality standards and controls, incremental loading, Change Data Capture (CDC), and error handling to guarantee reliability and accuracy.

• Create bronze tables and curated silver and gold tables by transforming, cleansing, and conforming raw data, while documenting lineage and transformations.

• Enhance performance through partitioning, clustering, and tuning of MS Fabric, SQL, and Spark workloads, while monitoring pipeline health and costs.

• Collaborate with Data Architects, Modellers, and Data Business Analysts to design schemas, naming conventions, and integration patterns that facilitate business self-service enablement through Gold data.

• Construct reusable components, templates, and automation scripts using Fabric Dataflows Gen2, Spark notebooks, or equivalent tools.

• Maintain metadata and sensitivity labels through Purview, ensuring appropriate access controls and compliance.

• Investigate and assess new features and patterns in Microsoft Fabric, providing recommendations for adoption to support an AI-driven data strategy.

• Guarantee that non-functional requirements such as resilience, availability, and security standards are adhered to according to agreed-upon metrics.

⛳️ Requirements

• 8+ years of experience in data engineering or data integration roles, with proficiency in SQL, Python, and/or Scala.

• Familiarity with ETL/ELT patterns, CDC, incremental loading, and data warehousing principles utilizing Microsoft Fabric.

• Hands-on experience with Delta Lake, Lakehouse/Medallion architecture, and cloud data platforms (Azure and AWS).

• Skilled in building pipelines with tools such as MS Fabric Dataflows Gen2, Spark/SQL notebooks, or equivalent platforms.

• Knowledgeable in data quality frameworks and monitoring/observability tools.

• Ability to collaborate effectively across teams and convey technical concepts to non-engineers.

• Experience with streaming data (Kafka, Event Hubs) or real-time ingestion.

• Understanding of master data management and data governance practices.

• Familiarity with version control and CI/CD for data pipelines.

• Previous experience in the implementation of governance, controls, and standards.

• Any prior experience in the Shipping or Maritime sector and Big Data Ecosystem is advantageous.

• Bachelor’s degree in computer science, engineering, or a related field.

• Certification in Azure Data Engineering is required.

• Additional certifications in AWS cloud data platform or other data platforms are beneficial.

🏝️ Benefits

• Working in a diverse society, we acknowledge that our customers, colleagues, and contractors are integral to our success.