Senior Data Engineer

This is a fully remote position, open to applicants in South Africa.

📋 Description

• Create and develop innovative data products and platforms utilizing high-volume telematics and IoT data on Snowflake and AWS.

• Design and oversee high-volume batch and event-driven processes for ingestion, transformation, and integration across SQL Server, Snowflake, and AWS.

• Implement incremental loading, change tracking, deduplication, handling of late-arriving data, and reconciliation processes.

• Ensure that retries, recovery, and backfilling processes are safe and repeatable.

• Investigate any instances of failed or delayed processing, including blocking, deadlocks, and backlogs.

• Identify root causes of issues and enhance reliability and throughput.

• Optimize SQL queries, stored procedures, execution plans, indexing, partitioning, and concurrent processing.

• Utilize Snowflake query profiles, pruning, clustering, and warehouse configurations to boost efficiency.

• Design maintainable data models and integration patterns.

• Write and review production code, automate deployment processes, and validate data integrity, performance, and concurrency.

• Define procedures for rollback and recovery.

• Set up monitoring and alerts for failures, data freshness, completeness, backlog, and throughput.

• Maintain documentation for pipelines, dependency maps, runbooks, and support handovers.

• Translate business needs into actionable solutions while coordinating priorities and dependencies.

• Mentor fellow engineers, review design proposals, and contribute to team development and recruitment efforts.

• Enhance testing, security measures, access controls, and development standards.

• Implement AI-assisted development and automation, accompanied by human review, testing, and appropriate data protection protocols.

• Collaborate with data engineers, BI specialists, and product teams to influence technical decisions and deliver dependable solutions.

⛳️ Requirements

• A relevant degree or tertiary qualification in Computer Science, Engineering, Information Technology, Data Engineering, or a related field, or equivalent practical professional experience.

• 4–8 years of hands-on experience in Data Engineering, including building and supporting production data pipelines.

• Strong practical experience working with SQL and Python in live production environments.

• Demonstrated experience with modern cloud and data platforms, particularly AWS and Snowflake.

• Solid understanding of data modeling, database design, performance optimization, and data engineering best practices.

• Familiarity with Git, code reviews, testing, and CI/CD practices.

• Strong analytical and problem-solving abilities for complex data and production challenges.

• Experience in mentoring engineers, participating in technical discussions, or reviewing technical designs is advantageous.

• Knowledge of Microsoft Azure is an advantage.

• Experience with telematics, IoT, time-series, or high-volume operational data is beneficial.

• Familiarity with multi-tenant systems and customer-facing reporting is beneficial.

• Experience with Kafka, Kinesis, Spark/PySpark, Iceberg, Delta Lake, or Hudi is beneficial.

• Experience with Airflow, dbt, and automated database deployment tools is beneficial.

• Experience with SQL Server tools such as T-SQL, SSIS, and SQL Agent is beneficial.

• Experience with C#, API integration or maintenance, Redshift, or other cloud data platforms is beneficial.

• Experience with data lineage, catalogs, data contracts, and practical governance is beneficial.

• Familiarity with GDPR or POPIA compliance requirements is advantageous.

🏝️ Benefits

• Equal opportunity employer dedicated to employment equity, diversity, and inclusion.

• Reasonable accommodations available throughout the recruitment process.

• People-first, people-centric culture.

• Opportunity to work on modern cloud and data platforms at IoT scale.

• Access to mentoring, technical guidance, and shared ownership opportunities.