Senior Data Engineer

Posted Nov 12

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and enhance scalable data pipelines utilizing Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.

• Conceptualize and execute Bronze, Silver, and Gold medallion architecture layers.

• Establish and sustain integrations with enterprise applications, REST APIs, databases, and various source systems.

• Create reusable frameworks for data ingestion to facilitate the onboarding of new data sources.

• Develop batch and near-real-time data ingestion pipelines featuring incremental loads, CDC, schema evolution, and error recovery.

• Configure and oversee Databricks Workflows, implementing pipelines through Auto Loader and Lakeflow Spark Declarative Pipelines.

• Enhance Spark and Delta Lake workloads by optimizing query performance, partitioning, joins, data-skew management, and computing costs.

• Enforce Unity Catalog governance, including access controls, lineage tracking, auditing, data masking, and row-level security.

• Manage data workflows on Amazon Redshift and Amazon EC2.

• Collaborate with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational challenges.

• Adhere to best practices for version control, code review, automated testing, CI/CD, and technical documentation.

• Partner with Data Analytics and business stakeholders to deliver reliable, well-documented, analytics-ready datasets.

• Provide technical direction and enhance automation, engineering standards, platform reliability, and resource utilization.

• Report directly to the Senior Manager - Business Insights & Analytics while collaborating with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders.


⛳️ Requirements

• Over 5 years of experience in data engineering or a similar technical role, encompassing the development and support of production data pipelines.

• Extensive hands-on knowledge of Databricks, particularly in developing, troubleshooting, and optimizing production workloads.

• Strong command of Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation.

• Practical experience with Delta Lake, including table management, incremental processing, and performance enhancement.

• Familiarity with data architecture and modeling, including medallion architecture and the creation of analytics-ready datasets.

• Experience with Databricks Workflows or similar orchestration solutions.

• Proven track record in building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications.

• Understanding of reliable data processing patterns, such as retries, idempotency, schema evolution, backfills, and failure recovery.

• Working knowledge of AWS services and cloud infrastructure, especially IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms.

• Familiarity with Git/GitHub, code reviews, automated testing, and CI/CD methodologies.

• Strong problem-solving abilities with the capacity to independently design, develop, deploy, and maintain production data solutions.

• Effective communication and collaboration skills for working efficiently with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders.

• Experience with Databricks Lakeflow Spark Declarative Pipelines, Auto Loader, and advanced incremental ingestion techniques is a plus.

• Experience implementing Unity Catalog governance is a plus.

• Familiarity with Databricks Genie, Genie Agents, and Genie Code is advantageous.

• Experience in AI/ML, Data Science, LLMs, contextualization, and Amazon Bedrock is a plus.


🏝️ Benefits

• Flexible remote work arrangement.

• AI-powered Talent Matching tool utilized in recruitment processes.

• Provision of privacy and data protection compliance information.

People also viewed

Trax Technologies9 hours ago

Data Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Fiserv10 hours ago

Senior Snowflake Data Platform and Artificial Intelligence Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Ontrac Solutions10 hours ago

Staff Data Engineer

US flagUnited States OnlyFreelanceData Engineer
ApplyView job
General Dynamics Information Technology10 hours ago

ICAM Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology10 hours ago

Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology10 hours ago

Senior Data Engineer

US flagUnited States OnlyFull-timeData Engineer$102k – $138k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers