Senior Data Engineer, Spark

This is a fully remote position, open to applicants in Egypt.

📋 Description

• Design and operate large-scale batch data pipelines, ensuring efficiency and cost-effectiveness as data volumes increase.

• Develop and oversee extensive data processing tasks utilizing Apache Spark.

• Conduct performance optimization to ensure smooth and dependable pipeline operations.

• Assist in data modeling for analytical purposes.

• Structure data within a lakehouse to enhance downstream accessibility.

• Manage automated scheduling, monitoring, and data quality verification.

• Collaborate with platform and product teams to facilitate comprehensive data flows.

⛳️ Requirements

• Proficient in hands-on Apache Spark, including performance optimization, rather than merely using it through a managed notebook.

• Experience in data modeling for analytics and organizing data within a lakehouse for downstream usability.

• Familiarity with automated scheduling, monitoring, and data quality verification.

• Ability to work with platform and product teams on comprehensive data flows.

• Knowledge of Apache Iceberg or other open table formats.

• Experience with Trino or similar query engines.

• Background in on-premises or self-managed cluster environments.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Opportunities for professional growth and development.

• Flexible work hours and remote working options.

• Comprehensive health and wellness benefits.

• Collaborative and inclusive work environment.