Lead - Data Engineering

This is a fully remote position, open to applicants in India.

📋 Description

• Assist in the support, maintenance, and enhancement of PDS data products and ETL pipelines utilizing DBT, AWS Glue, Databricks, and other cloud-native technologies.

• Analyze, troubleshoot, and resolve production incidents while ensuring high levels of availability, reliability, and adherence to SLAs.

• Create enhancements for pipelines, implement bug fixes, and develop operational automation to boost performance, scalability, and efficiency.

• Maintain data quality, integrity, and operational stability through validation, monitoring, root cause analysis, and proactive issue resolution.

• Collaborate with the Cloud Engineering team on managing cloud infrastructure.

• Work in conjunction with Analytics Engineering, Cloud Engineering, and Business teams on feature releases, production deployments, and continuous improvement initiatives.

• Develop and sustain monitoring systems, alerting mechanisms, dashboards, and operational documentation.

• Keep abreast of advancements in Databricks, Agentic AI, Retrieval-Augmented Generation, and AI-assisted development technologies to promote operational excellence.

⛳️ Requirements

• 3–5 years of experience in data engineering, production support, data operations, or a related field.

• Practical experience with AWS Glue, Databricks, DBT, and cloud-based data engineering services.

• Strong expertise in SQL, Python, and Apache Spark, along with experience in debugging and performance optimization.

• Solid understanding of ETL/ELT pipelines, data warehousing, data lakes, and distributed data processing.

• Familiarity with Apache Airflow, AWS Step Functions, or Databricks Workflows.

• Experience in production monitoring, incident management, root cause analysis, SLA management, and troubleshooting.

• Knowledge of data quality frameworks, validation processes, and data reconciliation.

• Proficiency in Git or SVN and Agile development methodologies.

• Familiarity with AWS services such as Amazon S3, Glue, Athena, Redshift, and CloudWatch.

• Working knowledge of Redshift, PostgreSQL, Athena, and other relational or analytical databases.

• Familiarity with tools like Jira, Confluence, and ServiceNow.

• Understanding of CI/CD practices, deployment processes, and production change management.

• Familiarity with Grafana or similar monitoring platforms.

• Knowledge of data governance, access management, IAM, data security, and compliance.

• Experience in building reusable automation frameworks, self-healing pipelines, or operational dashboards.

• Exposure to pharmaceutical or life sciences data platforms is a plus.

• Strong analytical thinking, problem-solving, and debugging skills.

• Excellent communication and presentation abilities.

• Proactive approach to operational enhancements, automation, and innovative solutions.

• Strong sense of ownership, accountability, and capability to work independently.

• Willingness to learn new technologies and collaborate across teams.

• Ability to prioritize multiple tasks and manage incidents in a fast-paced environment.

🏝️ Benefits

• Equal employment opportunity regardless of protected characteristics.

• Reasonable accommodations for qualified individuals with disabilities.