Senior Engineer – Data Engineering, Databricks

Posted Nov 12

This is a fully remote position, open to applicants in India.

📋 Description

• Provide technical guidance with minimal supervision to deliver solutions on the Databricks Platform.

• Develop data platforms utilizing cloud data lakehouse methodologies, distributed processing engines, event streaming platforms, and contemporary data technologies.

• Engage in the Databricks Center of Excellence.

• Construct next-generation data platforms for organizations focused on data and analytics.

• Collaborate with a Solution Architect to design and implement components of client data platform solutions.

• Take part in design sessions, decompose complex development tasks, and ensure timely completion of development items.

• Contribute to Center of Excellence initiatives by creating Databricks solution accelerators.

• Design and implement data solutions that leverage Databricks platform features.

• Work alongside cross-functional teams to design and optimize batch and streaming data pipelines.

• Guarantee data quality and pipeline efficiency.

• Remain up-to-date with emerging technologies and features of the Databricks platform.

• Continuously enhance skills related to Databricks and associated tools.


⛳️ Requirements

• Over 5 years of experience in data engineering.

• At least 2 years of practical experience in designing and developing data pipelines using Databricks.

• Familiarity with Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.

• Expertise in designing and constructing robust, scalable, YAML configuration-driven data pipelines.

• Experience with batch, micro-batch, and streaming data ingestion and processing patterns, utilizing tools like Auto Loader.

• Proficiency in building complex job/workflow orchestration patterns using Databricks Jobs/Workflows or tools like Airflow and dbt.

• Knowledge of data quality, pipeline audit, and observability solutions using Databricks-native features or frameworks such as Great Expectations, Collibra, and dbt.

• Proficient in Apache Spark, Presto, and Amazon EMR.

• Experience with Amazon Redshift, Snowflake, and Google BigQuery.

• Strong programming capabilities in SQL, stored procedures, and object-oriented programming languages such as Java, Python, and PySpark.

• Familiarity with DevOps and CI/CD pipelines using Databricks Asset Bundles, including automated validation and testing.

• Exposure to Infrastructure as Code tools like Terraform is a plus.

• Knowledge of NoSQL databases and container management systems.

• Familiarity with AI/ML tools such as MLflow, prompt engineering, and modern data and AI agentic workflows.

• Ideally, hold Databricks Data Engineering Associate and/or Professional certification.

• Experience delivering multiple Databricks projects.


🏝️ Benefits

• Contract engagement for a duration of 6 or more months.

• Anticipated start date in December or January.

People also viewed

Trax Technologies9 hours ago

Data Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Fiserv9 hours ago

Senior Snowflake Data Platform and Artificial Intelligence Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Ontrac Solutions10 hours ago

Staff Data Engineer

US flagUnited States OnlyFreelanceData Engineer
ApplyView job
General Dynamics Information Technology10 hours ago

ICAM Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology10 hours ago

Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology10 hours ago

Senior Data Engineer

US flagUnited States OnlyFull-timeData Engineer$102k – $138k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers