Senior Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and maintain a scalable data platform utilizing Databricks and AWS, adhering to Medallion (Bronze/Silver/Gold) architectural principles.

• Create and oversee data ingestion and ETL/ELT pipelines from essential business systems, guaranteeing data quality, reliability, and performance.

• Implement monitoring for data quality (freshness, completeness, validity, anomaly detection) complete with automated alerting systems.

• Establish and uphold Golden Record identity resolution and reverse synchronization across source systems.

• Manage metadata, lineage, and access permissions through Databricks Unity Catalog and platform standards.

• Collaborate with cross-functional teams to produce analytics- and AI-ready datasets that align with business goals.

• Apply best practices for infrastructure as code, continuous integration/continuous deployment (CI/CD), and version control for data pipelines.

• Communicate effectively with stakeholders, navigate ambiguity, and drive results in a dynamic, fast-paced environment.

⛳️ Requirements

• Over 8 years of experience in data engineering, specifically in building production-grade ETL/ELT pipelines at scale.

• Strong expertise in Python (including PySpark, scripting, API integrations, and pipeline automation).

• Practical experience with Databricks (notebooks, jobs, workflows, Unity Catalog) and AWS (Lambda, ECS/Fargate, S3, Glue, Athena, DynamoDB, EventBridge, Step Functions, IAM).

• In-depth knowledge of data modeling and lakehouse architectures (Medallion, Delta Lake, Parquet, schema evolution, incremental upserts).

• Proficient in SQL and dbt for developing modular, tested, and well-documented models.

• Experience with REST API integration, entity resolution, and record matching techniques.

• Familiarity with Git-based version control and CI/CD practices for data pipelines.

🏝️ Benefits

• Competitive benefits package

• Bonus opportunities

• Generous paid time off

• Paid holidays

• Programs designed to support employee well-being and work-life balance