Data Platform Engineer III

This is a fully remote position, open to applicants in Egypt.

📋 Description

• Design, implement, manage, and enhance extensive data platforms for both transactional and analytical functions.

• Create and sustain dependable batch and real-time ETL/ELT pipelines.

• Develop and manage CDC platforms utilizing Debezium, Kafka Connect, or comparable technologies.

• Deploy, administer, and optimize analytical databases such as StarRocks, ClickHouse, Apache Doris, or similar OLAP systems.

• Create scalable Kafka architectures that encompass topics, partitions, replication, consumer groups, and streaming pipelines.

• Manage MySQL and PostgreSQL clusters, including tasks like replication, backup, recovery, disaster recovery, and performance tuning.

• Enhance distributed query performance, storage configurations, indexing methods, and data lifecycle management.

• Facilitate large-scale data ingestion, transformation, and analytical operations while ensuring reliability and scalability.

• Design and maintain streaming and real-time data platforms using technologies like Kafka, Flink, Spark, or similar tools.

• Optimize distributed systems for throughput, latency, scalability, and resilience.

• Diagnose complex issues across distributed databases, messaging systems, and data processing pipelines.

• Collaborate with teams in Data Engineering, Analytics, and Data Science.

• Automate the provisioning, deployment, upgrades, scaling, and lifecycle management of data platforms.

• Develop self-service functionalities for engineering teams.

• Enhance observability, monitoring, reliability, and operational excellence throughout the data platform.

⛳️ Requirements

• 4–7 years of experience in Data Platform Engineering, Database Engineering, Platform Engineering, or Data Infrastructure.

• Robust production experience with MySQL and PostgreSQL.

• Extensive expertise in operating Kafka in production settings.

• Significant experience in constructing ETL/ELT pipelines.

• Practical experience with StarRocks, ClickHouse, Apache Doris, or similar OLAP databases.

• Familiarity with Kafka, Flink, Spark Streaming, or Kafka Streams.

• Strong skills in SQL optimization and database performance tuning.

• Experience with CDC technologies like Debezium or Kafka Connect.

• Kubernetes experience in deploying and managing stateful data workloads.

• Familiarity with AWS, GCP, OCI, or Azure.

• Experience with Terraform, Helm, GitOps, and automation frameworks.

• Proficient scripting abilities in Python, Bash, or Go.

• Experience with Prometheus, Grafana, ELK/OpenSearch, or LGTM.

• Preferred: Experience with Apache Iceberg, Delta Lake, or Apache Hudi.

• Preferred: Knowledge of Trino, Presto, Pinot, or Druid.

• Preferred: Experience supporting Data Science and Analytics platforms.

• Preferred: Skills in designing modern data lakehouse architectures.

• Preferred: Contributions to open-source data platform technologies.

• Certifications in Cloud, Kubernetes, Kafka, or Database are a plus.

🏝️ Benefits

• Competitive compensation.

• Top-tier health insurance.

• Enabling culture.

• Freedom and responsibility in the role.

• Fun and dynamic workplace.

• Work alongside leading AI professionals.

• Inclusive and empowering workplace culture.