Senior Data Engineer – Databricks, Contractor

This is a fully remote position, open to applicants in California.

📋 Description

• Provide technical leadership with minimal supervision to deliver solutions on the Databricks Platform.

• Construct data platforms utilizing cloud data lakehouse methodologies, distributed processing engines, event streaming platforms, and contemporary data technologies.

• Engage in the Databricks Center of Excellence.

• Collaborate with innovative organizations on data and analytics projects.

• Operate under a Solution Architect to design and implement components of clients’ data platform solutions.

• Participate in design discussions, decompose complex development tasks, and finalize development items punctually.

• Contribute to Center of Excellence initiatives by creating Databricks solution accelerators.

• Develop and execute data solutions using Databricks, including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.

• Work alongside cross-functional teams to design and refine batch and streaming data pipelines for quality and efficiency.

• Remain updated with emerging technologies and features of the Databricks platform.

⛳️ Requirements

• A minimum of 2 years of practical experience in data pipeline design and development with Databricks.

• Familiarity with Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.

• Expertise in designing and constructing scalable YAML configuration-driven data pipelines.

• Experience with batch, micro-batch, and streaming data ingestion and processing patterns using tools like Auto Loader.

• Proficient in creating complex job/workflow orchestration patterns with Databricks Jobs/Workflows, Airflow, dbt, or similar tools.

• Exposure to data quality, pipeline audit, and observability solutions utilizing Databricks native features or tools such as Great Expectations, Collibra, and dbt.

• Proficient in Apache Spark, Presto, and Amazon EMR.

• Experience with Amazon Redshift, Snowflake, and Google BigQuery.

• Strong programming capabilities in SQL, stored procedures, and object-oriented languages including Java, Python, and PySpark.

• Familiarity with DevOps and CI/CD pipelines using Databricks Asset Bundles, automated validation, and testing.

• Exposure to Infrastructure as Code tools like Terraform is advantageous.

• Familiar with NoSQL databases and container management systems.

• Exposure to AI/ML tools such as MLflow, prompt engineering, and modern data and AI agentic workflows.

• Databricks Data Engineering Associate and/or Professional certification along with multiple Databricks project delivery experiences is preferred.

🏝️ Benefits

• Salary is aligned with experience based on department and role.

• Equal employment opportunities and a work environment free of unlawful discrimination and harassment.