Databricks Solution Architect

This is a fully remote position, open to applicants in United States.

📋 Description

• Spearhead the architecture and development of the Azure Databricks platform, encompassing Delta Lake, Unity Catalog, and medallion architecture.

• Set technical standards, security measures, and scalable data integration frameworks.

• Supervise platform performance, monitoring, operational readiness, and cost efficiency.

• Oversee the transition of platform ownership from external implementation partners.

• Design, construct, and uphold production data pipelines utilizing PySpark, SQL, and Databricks Workflows.

• Integrate data from Salesforce, Orion, Addepar, PARis, and other enterprise systems.

• Create data transformation, reconciliation, and quality assurance processes.

• Apply automated testing, deployment, and infrastructure management techniques.

• Develop data infrastructure that supports AI applications, intelligent agents, and advanced analytics.

• Aid in document ingestion, vector search, model serving, and retrieval-augmented generation.

• Collaborate with the AI Engineer and Data Management Lead to provide secure, scalable solutions.

• Act as the technical design authority for the enterprise data platform.

• Assess vendor solutions and oversee the deliverables of implementation partners.

• Collaborate with business and technology leaders to convert requirements into effective solutions.

• Mentor junior engineers and advocate for engineering best practices.

• Design, implement, and sustain a secure, scalable, and high-performing Databricks platform.

• Develop and enhance production data pipelines, ensuring data accuracy, system reliability, cost effectiveness, and prompt resolution of technical issues.

• Establish architectural standards, guide implementation partners, and work with cross-functional teams to deliver solutions that meet business goals and regulatory standards.

⛳️ Requirements

• Bachelor's degree in Data, Information Systems, Computer Science, Business Analytics, or a related field, or equivalent combination of education and relevant professional experience.

• Over 10 years of experience in data engineering, data architecture, or related fields.

• More than 4 years of experience designing and managing production Databricks environments.

• In-depth expertise in Azure Databricks, Unity Catalog, Delta Lake, and data architecture principles.

• Strong practical experience with PySpark, SQL, data pipelines, and data integration.

• Experience in supporting AI or machine learning workloads within Databricks.

• Knowledge of Azure security, identity management, Terraform, and CI/CD practices.

• Proven experience managing enterprise data platforms and external technology partners.

• Excellent communication skills with the ability to convey technical concepts to business stakeholders.

• Advanced degree in Information Systems, Data Analytics, or a related discipline is preferred.

• Certification as a Data Management Professional (CDMP), Databricks Data Engineer Associate or Professional, or similar industry certifications is preferred.

• Databricks Data Engineer Professional certification and additional credentials in platform administration or architecture are preferred.

• Experience in financial services, wealth management, or regulated data environments is preferred.

• Familiarity with Salesforce Financial Services Cloud, Orion, or Addepar is preferred.

• Experience with Power BI, Databricks AI/BI Genie, and Delta Sharing is preferred.

• Capability to perform sitting and keyboarding duties.

• Strong attention to detail, analytical skills, prioritization, interaction, and collaboration abilities.

• Occasional flexibility in work hours may be necessary.

🏝️ Benefits

• Fully remote work environment.

• Limited to no travel required.

• Home-based work setting.

• Potential flexibility in work hours for system implementations, production incidents, or critical business operations.