Senior Analytics Engineer

This is a fully remote position, open to applicants in Africa.

📋 Description

• Develop and sustain AWS infrastructure for the analytics platform, encompassing storage, compute, networking, data cataloging, secrets, and access management.

• Oversee infrastructure as code using Terraform.

• Manage the ClickHouse Cloud warehouse, including its configuration, user management, role assignments, and performance/capacity monitoring.

• Host and maintain self-managed Apache Superset and Apache Airflow tools.

• Contribute to architectural decisions regarding the analytics platform.

• Construct and maintain data ingestion pipelines from application databases, APIs, and operational systems utilizing AWS Glue, Airbyte, and AWS Lambda.

• Implement effective and recoverable incremental loading patterns.

• Develop and optimize Spark jobs that process billions of device events.

• Model program contexts, including schools, terms, device allocations, and enrollments.

• Design and sustain Apache Iceberg tables for enhanced performance and cost efficiency.

• Validate data during ingestion and investigate any quality issues.

• Construct and maintain Airflow workflows that include dependencies, retries, and alerting mechanisms.

• Develop dbt staging and core models within ClickHouse.

• Implement incremental models that balance freshness, accuracy, and cost.

• Maintain dbt tests and model contracts.

• Document core models, their lineage, and approximations.

• Collaborate with the Senior BI & Data Analyst and Data Scientist.

• Construct CI/CD pipelines using GitHub Actions.

• Implement monitoring and alerting systems across pipelines, warehouses, and hosted tools.

• Respond to incidents, conduct root cause analyses, and support data quality processes.

• Maintain access controls and security protocols.

• Monitor AWS and ClickHouse expenses and optimize workloads accordingly.

• Maintain and test backup and recovery strategies.

• Take part in code reviews and produce runbooks and technical documentation.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Software Engineering, or a related field, or equivalent practical experience; demonstrated experience is considered equally important as formal credentials.

• A minimum of 7 years in data engineering, software engineering, or platform engineering roles.

• At least 3 years of experience in building and operating distributed data processing (Spark or similar) in a production environment.

• Proven responsibility for a production data platform or significant components, including reliability, performance, and daily operations.

• Experience handling datasets ranging from hundreds of millions to billions of records.

• Familiarity with making trade-offs between compute, storage, and performance within real budget constraints.

• Experience managing cloud infrastructure for data workloads.

• Proven capability to work in small teams with minimal supervision.

• Experience collaborating with remote teams across various countries and time zones.

• Proficient in Python with an emphasis on testing and modular design.

• Advanced SQL skills for analytical databases.

• Proficient in Spark (PySpark), including performance tuning for partitioning, shuffles, and data skew.

• Understanding of data lake architecture, including Parquet, partitioning strategies, and lake versus warehouse processing.

• Hands-on experience with AWS, including S3, Glue or EMR, IAM, and EC2.

• Proficient in infrastructure as code using Terraform, CloudFormation, CDK, or similar tools.

• Experience in Linux administration and Docker.

• Familiarity with dbt in production, including incremental models and testing.

• Experience in workflow orchestration with Airflow, Dagster, Prefect, or similar tools.

• Knowledge of Git, CI/CD, and code review practices.

• Strongly preferred: experience with Apache Iceberg or equivalent, ClickHouse or comparable data warehouse, Terraform, Apache Airflow, AWS Glue, Lake Formation, and analytics data modeling.

• Advantageous: experience with Databricks, Kafka/Flink/Kinesis/Spark Structured Streaming, event or IoT data, Airbyte, Apache Superset administration, and API or systems integration development.

🏝️ Benefits

• Full-time remote position.

• Occasional international travel for in-person team meetings, field support, or training as needed.

• Equal opportunity employment.