Databricks Practice Lead – Engineering Manager

This is a fully remote position, open to applicants in United States.

📋 Description

• Act as the organization's expert on the Databricks Lakehouse Platform.

• Create scalable, secure, and highly available data architectures utilizing Databricks, Apache Spark, Delta Lake, and cloud-native technologies.

• Direct batch, streaming, ETL, ELT, analytics, machine learning, and AI-driven data solutions.

• Set architectural standards for medallion architectures, data modeling, ingestion, transformation, orchestration, and data consumption.

• Establish governance frameworks for Unity Catalog, addressing lineage, access controls, auditing, metadata management, and secure data sharing.

• Provide guidance on workspace design, cluster configuration, serverless computing, workload isolation, performance tuning, and cost optimization.

• Supervise the integration of Databricks with Microsoft Azure, AWS, or Google Cloud.

• Develop or assess solutions involving PySpark, Spark SQL, Python, Delta Live Tables, Structured Streaming, Auto Loader, MLflow, and Databricks Workflows.

• Lead platform migrations and modernization efforts from legacy databases, data warehouses, Hadoop, and traditional ETL platforms.

• Set standards for source control, automated testing, CI/CD, infrastructure as code, monitoring, and production support.

• Conduct architecture reviews, code reviews, technical assessments, and root cause analyses.

• Manage, mentor, and nurture Databricks engineers, data engineers, architects, and technical consultants.

• Allocate resources, set goals and performance expectations, conduct reviews and coaching, and assist with recruiting and workforce planning.

• Create reusable accelerators, reference architectures, templates, and delivery playbooks.

• Develop and maintain a Databricks practice that supports multiple concurrent client engagements.

• Oversee project delivery from planning through implementation and operational support.

• Translate business, functional, security, and contractual requirements into technical plans and deliverables.

• Generate estimates, staffing plans, schedules, milestones, and strategies for risk mitigation.

• Monitor scope, schedule, quality, budget, resource utilization, dependencies, and technical risks.

• Coordinate efforts across engineering, cloud, cybersecurity, governance, analytics, project management, and client teams.

• Track delivery metrics and provide status updates to leadership, clients, and stakeholders.

• Lead technical escalations and support statements of work, proposals, estimates, presentations, demonstrations, and client meetings.

⛳️ Requirements

• Bachelor's degree in computer science, information technology, data engineering, engineering, or a related field.

• Minimum of 10 years of experience in data engineering, data architecture, analytics engineering, or similar technology roles.

• At least 5 years of hands-on experience in designing and implementing solutions using Databricks.

• A minimum of 3 years' experience managing or leading technical engineering teams.

• Advanced knowledge of Databricks Lakehouse Platform, Apache Spark, PySpark, Spark SQL, advanced SQL development, Delta Lake, medallion architecture, Unity Catalog, enterprise data governance, ETL and ELT pipeline architecture, batch and real-time data processing, data modeling, data warehousing, Python-based data engineering, Databricks Workflows, Jobs, and cluster management.

• Experience deploying Databricks solutions on Azure, AWS, or Google Cloud.

• Familiarity with CI/CD, Git-based development, automated testing, and infrastructure as code.

• Proven ability to optimize Spark workloads, cluster configurations, query performance, reliability, and cloud costs.

• Experience in managing technical delivery, resource assignments, risks, schedules, and client expectations.

• Excellent written, verbal, presentation, documentation, and stakeholder management skills.

• Ability to communicate complex technical concepts to executives, business stakeholders, and non-technical audiences.

• Preferred: Databricks or cloud professional certifications; experience in consulting, professional services, systems integration, or managed services; knowledge of government-client interactions; familiarity with federal security, privacy, governance, and compliance; experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, Terraform, MLflow, MLOps, generative AI, Databricks Mosaic AI, vector search, or machine learning deployment; knowledge of data standards and metadata frameworks; experience managing geographically distributed or remote teams; proposal and statement-of-work experience.

🏝️ Benefits

• No benefits, perks, or additional compensation details are specified in the posting.