Data Platform Engineer

This is a fully remote position, open to applicants in Romania, +1 more country.

📋 Description

• Develop and manage change-data-capture pipelines from PostgreSQL to Azure utilizing Kafka Connect and Debezium.

• Set up, deploy, and scale connectors comprehensively, managing connector configuration, task operations, offsets, schema history, and snapshot strategies.

• Operate pipelines as stateful workloads on Kubernetes (AKS), encompassing configuration, secrets management, networking, and resource optimization.

• Oversee and troubleshoot the platform in a production environment, addressing connector failures, task rebalances, restarts, throughput issues, backpressure, message-size limitations, retries, and recovery processes.

• Automate platform operations using Python by establishing configuration-driven onboarding, pipeline orchestration, monitoring and alerting systems, recovery workflows, and automated testing.

• Integrate CDC streams with Azure Event Hubs, ADLS, Azure PostgreSQL, ADF, and Databricks.

• Manage platform infrastructure as code to ensure environments are reproducible and modifications are subject to review.

• Implement data protection measures for sensitive information traversing the pipelines, including masking, hashing, access control, and retention policies.

⛳️ Requirements

• Proven commercial experience as a data or platform engineer, specifically with hands-on engagement in streaming or CDC pipelines rather than solely batch reporting.

• Solid understanding of Kafka concepts, including topics, partitions, offsets, consumer groups, and delivery semantics.

• Successful independent deployment of at least one Kafka Connect instance.

• Proficient SQL and PostgreSQL capabilities.

• Familiarity with WAL, logical replication, replication slots, and replication lag.

• Understanding of CDC principles: initial snapshots, inserts, updates, deletes, event ordering, at-least-once delivery, and schema evolution.

• Competent Python skills for automation and tool development.

• Practical experience with Azure data services, such as Event Hubs, ADLS, or Azure PostgreSQL.

• Comfortable using Kubernetes, involving workload deployment, configuration and secrets management, log reading, and debugging failing pods.

• Capability to debug active pipelines using metrics and logs.

• Production experience specifically with Debezium.

• Experience managing stateful workloads on AKS, including StatefulSets, stable worker identity, and resource tuning under load.

• Familiarity with infrastructure-as-code and CI/CD practices for data platform components, utilizing Terraform, Bicep, or similar tools.

• Practical experience with Databricks and ADF at a production level.

• Experience in implementing data protection controls for sensitive information, including masking, hashing, access control, and retention policies.

🏝️ Benefits

• Flexible employment options and remote work opportunities.

• Engagement in international projects with prestigious global clients.

• Opportunities for international business travel.

• A non-corporate work environment.

• Language training programs.

• Access to internal and external training sessions.

• Private healthcare and insurance coverage.

• Multisport card benefits.

• Well-being initiatives aimed at employee health and satisfaction.