Databricks Architect

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and construct scalable data platforms utilizing Databricks, Apache Spark, Delta Lake, and SQL, generating solution architecture and technical design documentation.

• Develop Medallion Architecture (Bronze, Silver, Gold) for enterprise data operations sourced from multiple claims systems.

• Create and enhance production-grade ETL/ELT pipelines using Databricks Workflows, PySpark, and SQL.

• Collaborate with ADF+ADLS for data ingestion and existing kill-and-fill dimension patterns.

• Design and uphold data models and Delta tables, implementing schema evolution, time travel, partitioning, and optimization strategies.

• Architect solutions on Azure, ensuring integration with cloud storage and databases.

• Implement Unity Catalog, including access controls, data governance, lineage, and security measures.

• Convert claim-system data dictionaries into Unity Catalog tables for reporting and APIs.

• Define and implement performance and cost-optimization strategies across the platform.

• Collaborate with data engineers, analysts, and business stakeholders to transform requirements into technical designs.

• Establish CI/CD practices for Databricks notebooks, jobs, and infrastructure.

• Diagnose issues in production data pipelines and support platform reliability.

⛳️ Requirements

• 5–7 years of experience in data engineering or cloud environments.

• Strong practical experience with Databricks and Apache Spark.

• Proficient in PySpark and SQL at an advanced level.

• Deep understanding of Delta Lake and Medallion Architecture.

• Familiarity with at least one cloud platform, specifically Azure.

• Knowledge of Data Governance, Unity Catalog, IAM/RBAC, and security protocols.

• Experience with Git, CI/CD, and infrastructure-as-code methodologies.

• Solid understanding of data warehousing, dimensional modeling, and distributed data processing.

• Preferred: Experience with Azure Data Factory.

• Preferred: Knowledge of Terraform.

• Preferred: Familiarity with Kafka or other streaming technologies.

• Preferred: Experience with Power BI platforms.

• Preferred: Databricks certifications.

• Preferred: Background in designing enterprise data lakes/lakehouses.

🏝️ Benefits

• Comprehensive coverage for medical, dental, vision, and life insurance.

• 401(k) retirement plan with company matching.

• Health Savings Account (HSA) and Flexible Spending Accounts (FSAs).

• Paid time off (PTO) and disability leave.

• Employee Assistance Program (EAP).