Senior Engineer – Data Engineering, Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Act as a technical leader within teams that are focused on delivering innovative solutions using the Databricks Platform.

• Develop next-generation data platforms utilizing cloud data lakehouse methodologies, distributed processing engines, event streaming platforms, and contemporary data technologies.

• Engage in the Databricks Center of Excellence.

• Collaborate with a Solution Architect to design and implement components of client data platform solutions.

• Participate in design sessions, decompose complex development tasks, and ensure timely completion of development items.

• Contribute to Center of Excellence initiatives by creating Databricks solution accelerators.

• Design and implement data solutions leveraging Databricks.

• Work with cross-functional teams to design and enhance batch and streaming data pipelines, ensuring data quality and efficiency.

• Keep abreast of emerging technologies and features of the Databricks platform while continuously enhancing relevant skills.

⛳️ Requirements

• Over 5 years of experience in data engineering, with a minimum of 2 years of hands-on experience in designing and developing data pipelines using Databricks.

• Familiarity with Databricks platform features including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.

• Expertise in designing and constructing robust, scalable, YAML configuration-driven data pipelines that utilize batch, micro-batch, and streaming data ingestion and processing techniques with tools like Auto Loader.

• Experience in developing complex job/workflow orchestration patterns using Databricks jobs/workflows or similar tools such as Airflow and dbt.

• Knowledge of data quality, pipeline auditing, and observability solutions employing Databricks native features or tools/frameworks like Great Expectations, Collibra, and dbt.

• Proficiency in Apache Spark, Presto, and Amazon EMR.

• Experience with data warehousing solutions such as Amazon Redshift, Snowflake, and Google BigQuery.

• Strong programming abilities in SQL, stored procedures, and object-oriented programming languages such as Java, Python, and PySpark.

• Familiarity with DevOps practices and CI/CD pipelines utilizing Databricks Asset Bundles with automated validation and testing processes.

• Exposure to Infrastructure as Code tools like Terraform is highly advantageous.

• Understanding of NoSQL databases and container management systems.

• Experience with AI/ML tools such as MLflow, prompt engineering, and modern data and AI agentic workflows.

• Databricks Data Engineering Associate and/or Professional certification is preferred but not mandatory.

• Prior experience in delivering multiple Databricks projects is advantageous.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance.

• Generous retirement plan with employer matching.

• Flexible work hours and remote work opportunities.

• Professional development and training programs.

• A dynamic and collaborative work environment.