Azure Databricks Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, construct, and uphold scalable ETL/ELT pipelines utilizing Azure Databricks and Apache Spark;

• Create and enhance PySpark/Spark SQL notebooks for extensive data processing;

• Establish batch and real-time data ingestion pipelines through Azure Data Factory (ADF), Databricks Workflows, and Structured Streaming;

• Oversee and refine Delta Lake/Lakehouse architecture, encompassing Bronze, Silver, and Gold layers;

• Integrate data from various sources such as APIs, databases, cloud storage, and external systems;

• Collaborate with Azure services including ADLS Gen2, Azure Synapse Analytics, Azure SQL Database, and Azure DevOps;

• Monitor, troubleshoot, and enhance pipeline performance, scalability, and cost efficiency;

• Enforce data governance, security, access controls, and compliance standards utilizing Unity Catalog and RBAC;

• Work alongside Data Engineers, Data Scientists, Analysts, and Product teams to provide dependable data solutions;

• Assist in the implementation of CI/CD, automation, and infrastructure-as-code practices.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related discipline;

• Over 5 years of experience in Data Engineering or Big Data development;

• Strong practical experience with Azure Databricks and Apache Spark;

• Proficient in PySpark, Spark SQL, and SQL;

• Familiarity with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS);

• Understanding of Delta Lake, Lakehouse architecture, and data modeling principles;

• Experience in distributed data processing and performance tuning;

• Knowledge of CI/CD pipelines, Git, and Azure DevOps;

• NICE TO HAVE: Experience with Kafka, dbt, Airflow, or Snowflake. Familiarity with Terraform or Infrastructure as Code (IaC). Exposure to MLflow, machine learning pipelines, or MLOps.

🏝️ Benefits

• Digital-First Approach: Great talent knows no borders! You can work from wherever you are — we hire and collaborate with professionals worldwide.

• Remote Work Model: Balance your professional and personal life with our flexible working conditions, empowering you to deliver your best from anywhere.

• Exciting Projects: Dive into impactful projects across industries that challenge and spark creativity.

• Boost Your Expertise: Grow your career with continuous learning, development opportunities, and hands-on experience.

• Join the Best Team Ever: Collaborate with our diverse and cross-cultural team of passionate technologists and creative thinkers.