Data Engineer, Databricks

📋 Description

• Lead and actively participate in comprehensive Databricks implementations for clients, covering data migration, Lakehouse architecture, and pipeline development.

• Collect technical requirements, design solutions, and present recommendations to both technical and business stakeholders of clients.

• Construct scalable ETL/ELT pipelines utilizing PySpark, Delta Lake, Delta Live Tables (DLT), and Databricks Workflows.

• Design and implement Databricks Genie.

• Create and establish semantic layers.

• Leverage Databricks AI capabilities to enhance development, debugging, and code optimization.

• Design and implement Lakebase architectures for both operational and analytical workloads, including transactional data use cases.

• Develop solutions adhering to SDLC best practices, such as modular code design, testing, and documentation.

• Utilize Git-based version control with appropriate branching strategies.

• Implement CI/CD pipelines for Databricks assets.

• Enforce data quality checks, validations, and expectations within workflows.

• Design and implement governance, security, and lineage solutions using Unity Catalog.

• Optimize Databricks workloads for performance, cost, and reliability using tools such as Photon, cluster policies, Liquid Clustering, and Auto Loader.

• Integrate Databricks with client ecosystems, including Azure, AWS, GCP, Snowflake, Kafka, and legacy systems.

• Support client workshops, proof-of-concepts (POCs), and knowledge transfer sessions.

• Deliver projects using consulting methodologies while adhering to quality, timeline, and budget expectations.

• Document architectures, runbooks, and best practices for client reference.

• Engage in solutioning activities, including scoping, estimation, and technical demonstrations as required.

⛳️ Requirements

• 3-5 years of practical experience with Databricks (or substantial recent experience with Spark and Databricks).

• Demonstrated experience in delivering Databricks projects within a consulting or professional services context (preferred) or equivalent client-facing project experience.

• Strong expertise in PySpark, Spark SQL, Python, and SQL.

• Extensive experience with Delta Lake, Unity Catalog, Delta Live Tables, and Databricks Jobs.

• Practical experience with Git version control, including pull requests, code reviews, and collaborative development workflows.

• Cloud platform experience (Azure Databricks, AWS, or GCP - at least one).

• Excellent client-facing and communication skills, capable of explaining complex concepts to both technical and non-technical audiences.

• Solid understanding of data governance, security, and Lakehouse best practices.

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).

🏝️ Benefits

• As a Databricks Data Engineer, you will directly engage with clients from various industries to design, implement, and enhance Databricks-based data solutions.

• You will play a crucial role in our Professional Services delivery teams, ensuring the timely and within-scope delivery of high-quality projects while fostering strong client relationships.

• This position is client-facing and merges hands-on technical delivery with consulting best practices.

• Collaborate with client data teams to facilitate successful adoption and handover of solutions.