Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• ETL/ELT Pipeline Development: Construct and maintain scalable data pipelines utilizing AWS. Implement both batch and incremental loading strategies to meet BI reporting and application data requirements.

• Real-Time Data Streaming: Design and oversee real-time data ingestion pipelines using Kafka. Ensure a low-latency, fault-tolerant data flow essential for critical business operations.

• Workflow Orchestration: Create, schedule, and monitor comprehensive data workflows with Apache Airflow. Manage dependencies, retries, and alert systems for production DAGs.

• Data Warehouse Management: Manage and optimize Amazon Redshift clusters, including schema design, query performance tuning, distribution/sort keys, and vacuuming to guarantee high availability and cost efficiency.

• Data Quality & Observability: Implement automated checks for data quality during ingestion and transformation processes. Define validation rules, establish alerting for anomalies and discrepancies, and set SLAs to ensure stakeholders can rely on the data they utilize.

• API Integrations: Incorporate external and internal REST APIs into data pipelines to extract operational and product data for the warehouse.

• Cloud Cost Optimization: Oversee and right-size data processing and storage resources across S3, EMR, Redshift, EC2, and Lambda. Actively identify inefficiencies and suggest cost-saving enhancements.

• BI & Analytics Collaboration: Collaborate with the BI team to ensure data models, preprocessing logic, and Redshift schema design align with reporting and dashboard requirements.

⛳️ Requirements

• Bachelor’s degree in Computer Science or a related quantitative discipline.

• Over 2 years of experience as a Data Engineer.

• Strong proficiency in Python and SQL for data transformation and pipeline creation.

• Practical experience with Apache Spark (PySpark) for large-scale data processing.

• Familiarity with Kafka for real-time data ingestion and stream processing.

• Hands-on experience in managing and maintaining Airflow DAGs in production settings.

• Knowledge of Redshift performance tuning, schema design, and query optimization.

• Experience in implementing automated data validation and quality checks within pipelines.

• Detail-oriented with a strong interest in data transformations and their influence on business outcomes.

• Excellent problem-solving and time management abilities.

• Previous experience in project or team management is preferred, along with a passion for mentoring and guiding others.

🏝️ Benefits

• Professional growth in a dynamic, rapidly expanding, high-social-impact industry.

• A collaborative, open-minded culture comprised of enthusiastic colleagues driven by the challenge of innovation to create a significant impact on people and the planet.

• A truly multicultural experience: the opportunity to work with and learn from individuals from diverse geographies, nationalities, and backgrounds.

• Structured, tailored learning and development programs to enhance your leadership, management, and professional skills through the Sun King Center for Leadership.