Lead Technical Consultant – Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the technical execution of Databricks workstreams, encompassing requirements gathering, design, development, testing, deployment, and transition to support.

• Collect and assess business and technical requirements, converting them into actionable solution designs, implementation strategies, and technical specifications.

• Create and develop scalable data pipelines and lakehouse solutions utilizing Databricks, Apache Spark/PySpark, Python, SQL, and Delta Lake.

• Direct the deployment and integration of Databricks with cloud data platforms, source systems, data warehouses, analytics tools, and downstream applications.

• Configure and enhance Databricks jobs, workflows, clusters, notebooks, compute resources, and data pipelines to ensure performance, reliability, scalability, and cost-effectiveness.

• Enforce data governance, security, access controls, lineage, and compliance practices through Unity Catalog and associated cloud services.

• Set development standards and spearhead design reviews, code reviews, unit testing, documentation, version control, CI/CD, and troubleshooting within the designated workstream.

• Facilitate estimation processes, identify technical dependencies and risks, and assist in defining realistic development timelines and delivery objectives.

• Mentor and guide consultants and engineers, delegate technical responsibilities, and ensure the team consistently produces high-quality work.

• Collaborate with clients and cross-functional stakeholders, generate technical documentation, and lead solution walkthroughs, issue resolution, knowledge transfer, and enablement.

• Assess relevant Databricks features and emerging technologies, recommending practical enhancements that improve solution quality and delivery.

⛳️ Requirements

• A Bachelor's degree in Computer Science, Software Engineering, Information Technology, Data Engineering, or a related discipline; equivalent relevant experience may be accepted.

• 5-10 years of experience in technical consulting, data engineering, analytics engineering, software development, or similar technology positions.

• A minimum of 3 years of hands-on experience in designing, developing, and delivering solutions on the Databricks platform.

• Proven experience leading a technical workstream or a small delivery team on mid- to large-scale client implementations.

• Strong hands-on experience with Python, SQL, Apache Spark/PySpark, ETL/ELT frameworks, and the development of data pipelines.

• Familiarity with key Databricks features such as Delta Lake, Unity Catalog, Workflows/Jobs, notebooks, compute or cluster management, and platform administration concepts.

• Experience integrating Databricks with at least one major cloud platform: Microsoft Azure, Amazon Web Services (AWS), or Google Cloud Platform (GCP).

• In-depth understanding of lakehouse, data lake, data warehouse, batch and streaming processing, data modeling, and contemporary analytics principles.

• Experience with Git-based development, DevOps, automated testing, CI/CD, Agile delivery, and managing multiple priorities or workstreams.

• Client-facing consulting experience, including requirements discussions, technical workshops, solution presentations, estimates, and delivery communications.

• Preferred Certifications and Experience: Databricks Data Engineer Associate, Databricks Data Engineer Professional, or another relevant Databricks certification.

• Experience with MLflow, machine learning workflows, real-time or streaming solutions, or Databricks SQL.

• Experience supporting technical discovery, solution estimates, proposals, or statements of work in collaboration with architects and sales teams.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work arrangements and remote work options.

• Professional development opportunities and access to training resources.

• Collaborative work environment with a focus on innovation.