Senior Engineer – Data Engineering, Databricks

Posted Nov 12

This is a fully remote position, open to applicants in India.

📋 Description

• Deliver innovative solutions on the Databricks Platform while providing technical leadership with minimal supervision.

• Utilize cloud data lakehouse methodologies, distributed processing engines, event streaming platforms, and contemporary data technologies.

• Engage in the Databricks Center of Excellence.

• Construct next-generation data platforms for organizations focused on data and analytics.

• Collaborate with a Solution Architect to design and implement components of client data platform solutions.

• Take part in design sessions, decompose intricate development tasks, and ensure timely completion of development items.

• Contribute to initiatives within the Center of Excellence by creating Databricks solution accelerators.

• Develop and execute data solutions utilizing Databricks.

• Work alongside cross-functional teams to design and optimize both batch and streaming data pipelines.

• Maintain data quality and efficiency.

• Keep up with emerging technologies and features of the Databricks platform, continuously enhancing relevant skills.


⛳️ Requirements

• Over 5 years of experience in data engineering.

• A minimum of 2 years of practical experience in designing and developing data pipelines using Databricks.

• Familiarity with Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.

• Expertise in designing and constructing scalable YAML configuration-driven data pipelines.

• Experience with batch, micro-batch, and streaming data ingestion and processing patterns utilizing tools such as Auto Loader.

• Proven experience in developing complex job/workflow orchestration patterns using Databricks Jobs/Workflows or tools like Airflow and dbt.

• Knowledge of data quality, pipeline audit, and observability solutions using Databricks native features or frameworks/tools such as Great Expectations, Collibra, and dbt.

• Proficient in Apache Spark, Presto, and Amazon EMR.

• Experience with Amazon Redshift, Snowflake, and Google BigQuery.

• Strong programming abilities in SQL, stored procedures, and object-oriented programming languages like Java, Python, and PySpark.

• Understanding of DevOps and CI/CD pipelines using Databricks Asset Bundles, including automated validation and testing.

• Familiarity with Infrastructure as Code tools such as Terraform is advantageous.

• Knowledge of NoSQL databases and container management systems.

• Exposure to AI/ML tools such as MLflow, prompt engineering, and modern data and AI workflows.

• Databricks Data Engineering Associate and/or Professional certification is preferred.

• Experience delivering multiple projects using Databricks.

• A curious, self-motivated mindset with the ability to share insights with teams.


🏝️ Benefits

• Contract engagement lasting 6 months or more.

• Start date in December or January.

• Remote work arrangement.

People also viewed

Trax Technologies9 hours ago

Data Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Fiserv9 hours ago

Senior Snowflake Data Platform and Artificial Intelligence Architect

US flagUnited States OnlyFull-timeData Engineer
ApplyView job
Ontrac Solutions10 hours ago

Staff Data Engineer

US flagUnited States OnlyFreelanceData Engineer
ApplyView job
General Dynamics Information Technology10 hours ago

ICAM Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology10 hours ago

Identity Data Management Engineer – Radiant Logic

US flagUnited States OnlyFull-timeData Engineer$170k – $230k/year
ApplyView job
General Dynamics Information Technology10 hours ago

Senior Data Engineer

US flagUnited States OnlyFull-timeData Engineer$102k – $138k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers