Senior Databricks Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, construct, and sustain scalable ETL/ELT pipelines on the Databricks Lakehouse Platform utilizing PySpark, Spark SQL, and Delta Lake.

• Develop and implement medallion (bronze/silver/gold) data architectures while ensuring adherence to data quality, governance, and lineage standards.

• Enhance Spark jobs and cluster settings for optimal performance and cost efficiency, incorporating strategies such as partitioning, caching, and autoscaling.

• Execute and oversee Unity Catalog for managing access control, data governance, and asset sharing across workspaces.

• Create and manage workflows using Databricks Workflows, Delta Live Tables, and CI/CD pipelines.

• Partner with data scientists, analysts, and business stakeholders to convert requirements into dependable data products.

• Establish engineering best practices, perform code reviews, and guide junior data engineers.

• Monitor production pipelines, resolve issues, and lead root-cause analysis while fostering continuous improvement.

⛳️ Requirements

• A minimum of 5 years of experience in data engineering, including at least 3 years developing production solutions on Databricks and Apache Spark.

• Advanced proficiency in Python (PySpark) and SQL.

• Extensive hands-on experience with Delta Lake, Unity Catalog, and the medallion architecture pattern.

• Strong background in at least one major cloud platform (AWS, Azure, or GCP) along with its core data services.

• Demonstrated success in optimizing Spark performance and managing cluster costs.

• Experience in data modeling, warehousing concepts, and constructing dimensional/analytics-ready datasets.

• Proficient with Git-based version control, CI/CD, and infrastructure-as-code practices.

• Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.

🏝️ Benefits

• Bounteous is proud to be an equal opportunity employer.

• Bounteous does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, physical or mental disability, national origin, veteran status, or any other status protected under federal, state, or local law.

• Bounteous is willing to sponsor eligible candidates for employment visas.