
Senior Data Engineer
Posted Nov 12

Posted Nov 12
This is a fully remote position, open to applicants in India.
• Design, develop, and enhance scalable data pipelines utilizing Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.
• Conceptualize and execute Bronze, Silver, and Gold medallion architecture layers.
• Establish and sustain integrations with enterprise applications, REST APIs, databases, and various source systems.
• Create reusable frameworks for data ingestion to facilitate the onboarding of new data sources.
• Develop batch and near-real-time data ingestion pipelines featuring incremental loads, CDC, schema evolution, and error recovery.
• Configure and oversee Databricks Workflows, implementing pipelines through Auto Loader and Lakeflow Spark Declarative Pipelines.
• Enhance Spark and Delta Lake workloads by optimizing query performance, partitioning, joins, data-skew management, and computing costs.
• Enforce Unity Catalog governance, including access controls, lineage tracking, auditing, data masking, and row-level security.
• Manage data workflows on Amazon Redshift and Amazon EC2.
• Collaborate with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational challenges.
• Adhere to best practices for version control, code review, automated testing, CI/CD, and technical documentation.
• Partner with Data Analytics and business stakeholders to deliver reliable, well-documented, analytics-ready datasets.
• Provide technical direction and enhance automation, engineering standards, platform reliability, and resource utilization.
• Report directly to the Senior Manager - Business Insights & Analytics while collaborating with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders.
• Over 5 years of experience in data engineering or a similar technical role, encompassing the development and support of production data pipelines.
• Extensive hands-on knowledge of Databricks, particularly in developing, troubleshooting, and optimizing production workloads.
• Strong command of Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation.
• Practical experience with Delta Lake, including table management, incremental processing, and performance enhancement.
• Familiarity with data architecture and modeling, including medallion architecture and the creation of analytics-ready datasets.
• Experience with Databricks Workflows or similar orchestration solutions.
• Proven track record in building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications.
• Understanding of reliable data processing patterns, such as retries, idempotency, schema evolution, backfills, and failure recovery.
• Working knowledge of AWS services and cloud infrastructure, especially IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms.
• Familiarity with Git/GitHub, code reviews, automated testing, and CI/CD methodologies.
• Strong problem-solving abilities with the capacity to independently design, develop, deploy, and maintain production data solutions.
• Effective communication and collaboration skills for working efficiently with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders.
• Experience with Databricks Lakeflow Spark Declarative Pipelines, Auto Loader, and advanced incremental ingestion techniques is a plus.
• Experience implementing Unity Catalog governance is a plus.
• Familiarity with Databricks Genie, Genie Agents, and Genie Code is advantageous.
• Experience in AI/ML, Data Science, LLMs, contextualization, and Amazon Bedrock is a plus.
• Flexible remote work arrangement.
• AI-powered Talent Matching tool utilized in recruitment processes.
• Provision of privacy and data protection compliance information.
Trax Technologies
Fiserv
Ontrac Solutions
General Dynamics Information Technology
Get handpicked remote jobs straight to your inbox weekly.