Senior Data Engineer

This is a fully remote position, open to applicants in Netherlands, +3 more countries.

📋 Description

• Design and construct data pipelines that transfer information from terminal operating systems like Navis N4 to the Azure-based platform.

• Create the transformation layer that converts raw operational data into well-organized, documented datasets.

• Develop ELT pipelines from on-premises terminal databases to the Azure platform.

• Implement near-real-time microbatch refreshes every three minutes for operational reporting and daily batches for analytics.

• Utilize watermark-based incremental queries over Kafka/Azure Event Hubs to PostgreSQL staging.

• Build and enhance a multi-tenant Azure PostgreSQL architecture featuring database-per-tenant isolation.

• Collaborate with the STG → ODS → DataMart layer structure and the internal declarative transformation framework, dewi.

• Develop and improve internal Go connector tools.

• Implement data quality monitoring, which includes gap detection, freshness alerts, and reconciliation.

• Define the standardized terminal data model across various TOS versions and configurations.

• Engage directly with terminal operations personnel without intermediaries.

• Assist in scaling engineering foundations for Terminal.exe, the SaaS platform for container terminal operators offered by All Your BI.

⛳️ Requirements

• Over 5 years of experience in data engineering, encompassing complete ownership of production data platforms.

• Proficiency in SQL and Python.

• Experience with Go is a significant advantage.

• Strong background with SQL Server as a source system and PostgreSQL as a target storage solution.

• Demonstrated experience in building and managing ELT pipelines against operational source systems.

• Familiarity with watermark/incremental or CDC/replica-based extraction techniques.

• Knowledge of event-driven or streaming architectures.

• Experience with Azure cloud data platforms.

• Familiarity with AKS is a plus.

• Knowledge of Azure DevOps CI/CD is advantageous.

• Understanding of GitOps and Argo CD is beneficial.

• Strong emphasis on data accuracy and instrumentation.

• Experience with OpenTelemetry.

• Knowledge of the container terminal or port logistics industry is a plus.

• Direct experience with Navis N4 or another TOS is advantageous.

• Experience in multi-tenant SaaS data architecture is a plus.

• Familiarity with declarative, YAML-driven transformation frameworks is beneficial.

• Must be eligible to work in Europe.

🏝️ Benefits

• €5,000 annually for personal professional development.

• 25 vacation days plus the option to purchase up to 10 additional days.

• Local public holidays.

• Up to 4 months of sabbatical leave.

• Two international trips per year with the entire team.

• Occasional beverages and team outings.

• €500 budget for home office setup.

• Provision of a laptop.

• Fully remote work within Europe.

• Statutory benefits including pension, health, social security, leave, sick pay, and notice in accordance with the hire's country, either via local contract or EOR.