Resident Solution Architect, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and oversee the implementation of data platforms based on Databricks, utilizing best practices from Lakehouse and Medallion architectures.

• Deliver hands-on technical leadership across data engineering, analytics, and AI-driven use cases utilizing Databricks.

• Establish and uphold standards for data governance, security, and access control through Databricks Unity Catalog and principles of enterprise data management.

• Facilitate and support AI/ML workloads, including machine learning pipelines and model lifecycle integration within the Databricks environment.

• Enhance platform performance, scalability, and cost efficiency with a focus on FinOps principles.

• Work collaboratively with business leaders, architects, and engineering teams to ensure platform capabilities meet enterprise data and analytics requirements.

• Act as a resident technical expert, providing architectural guidance, mentoring teams, and ensuring the quality and consistency of solutions.

⛳️ Requirements

• Over 10 years of comprehensive experience in data engineering, data architecture, or analytics engineering roles.

• More than 5 years of hands-on experience with Databricks, covering Delta Lake, Unity Catalog, Workflows/Jobs, and Lakehouse architecture.

• Databricks Professional Certification (REQUIRED).

• Significant expertise in Apache Spark (with a preference for PySpark), Python, and advanced SQL.

• Demonstrated experience in designing and implementing production-level, large-scale data pipelines.

• Practical experience with at least one major cloud platform (preferably Azure, with AWS or GCP being acceptable).

• Knowledge in data modeling, Medallion architecture, and designing enterprise data platforms.

• Familiarity with CI/CD pipelines, infrastructure automation tools (such as Terraform), and Agile delivery methodologies.

• Excellent communication and stakeholder engagement skills, with experience in regulated enterprise environments.

🏝️ Benefits

• Health insurance

• Retirement plans

• Paid time off

• Flexible work arrangements

• Professional development