Databricks Solution Architect

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and lead the deployment of an enterprise lakehouse on Databricks (Delta Lake, Unity Catalog, Photon, Workflows) across one or multiple major cloud platforms (AWS, Azure, or GCP).

• Create scalable batch and streaming data pipelines utilizing PySpark, Spark SQL, Structured Streaming, and Delta Live Tables; establish best practices for data ingestion from operational systems, event streams, and third-party APIs.

• Set and uphold platform standards for data modeling (medallion architecture), CI/CD, code quality, testing, observability, and cost efficiency.

• Direct the governance strategy using Unity Catalog — implementing fine-grained access control, data lineage, auditing, and PII management — in collaboration with security and compliance teams.

• Enhance Spark workload performance and cost-effectiveness: including cluster sizing, Photon, autoscaling, file layout, Z-ordering, caching, and query tuning.

• Collaborate with ML engineers and data scientists to operationalize models using MLflow, feature stores, and model serving on Databricks.

• Manage the cloud infrastructure for the platform, encompassing networking, IAM, secrets, encryption, and Terraform/IaC for Databricks workspaces and related services.

• Mentor a team of data engineers; facilitate architecture reviews, code reviews, and technical design discussions; elevate engineering standards.

• Work with stakeholders across analytics, product, and finance to transform business requirements into a strategic roadmap for the data platform.

⛳️ Requirements

• Over 8 years of data engineering experience, with a minimum of 4 years developing production workloads on Databricks.

• Extensive knowledge of Apache Spark (PySpark and Spark SQL) — including performance optimization, partitioning strategies, and the Catalyst/Photon execution model.

• Significant hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.

• Practical experience on at least one major cloud provider (AWS, Azure, or GCP), including networking, IAM, storage (S3/ADLS/GCS), and compute resources.

• Proficient in Python and SQL; familiarity with Scala is advantageous.

• Experience in designing medallion (bronze/silver/gold) architectures and dimensional models for analytics.

• Strong CI/CD and DevOps practices: including Git, Terraform, Databricks Asset Bundles or dbx, and automated testing of data pipelines.

• Proven record of leading technical projects from inception to completion and mentoring engineering teams.

• Exceptional written and verbal communication skills; capable of fostering alignment between engineering and business stakeholders.

🏝️ Benefits

• Bounteous is proud to be an equal opportunity employer.

• Bounteous does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, physical or mental disability, national origin, veteran status, or any other status protected under federal, state, or local law.

• Bounteous is willing to sponsor eligible candidates for employment visas.