Senior Consultant, Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and maintain scalable data pipelines utilizing Databricks.

• Construct and enhance ETL/ELT workflows leveraging PySpark and Spark SQL.

• Create and oversee Delta Lake architecture for dependable and efficient data processing.

• Integrate data from various sources including databases, APIs, streaming platforms, and cloud storage.

• Collaborate with business partners, data analysts, and data scientists to comprehend business needs.

• Fine-tune Spark jobs for enhanced performance, scalability, and cost efficiency.

• Enforce data quality, governance, and security best practices.

• Monitor production workloads and resolve performance issues.

• Automate deployment processes through CI/CD pipelines.

• Guide junior team members and offer technical support during project execution.

⛳️ Requirements

• Significant experience with the **Databricks Platform**.

• Practical expertise in **Apache Spark**, **PySpark**, and **Spark SQL**.

• Familiarity with **Delta Lake**, Unity Catalog, and Databricks Workflows.

• Strong skills in SQL programming and data modeling.

• Experience with cloud platforms such as **AWS**, **Azure**, or **Google Cloud Platform (GCP)**.

• Experience in working with data lakes, data warehouses, and contemporary data architectures.

• Knowledge of orchestration tools like **Azure Data Factory (ADF)**, **Apache Airflow**, or similar.

• Proficiency with version control systems like Git.

• Understanding of CI/CD methodologies and DevOps principles.

• Exceptional analytical, troubleshooting, and problem-solving abilities.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and career growth.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.