Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Define and enhance data architectures leveraging Databricks, focusing on optimizing pipelines, enhancing processing efficiency, and addressing complex technical challenges.

• Construct and sustain scalable, dependable data pipelines utilizing contemporary distributed processing technologies to guarantee high-quality, efficient data ingestion, transformation, and delivery.

• Collaborate with Apache Spark, Airflow, Kafka, dbt, SQL/NoSQL databases (PostgreSQL, MongoDB, Cassandra), and cloud services (Databricks, AWS Glue, BigQuery).

• Design, outline, and execute scalable data architectures based on the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold).

• Implement batch and streaming data processing solutions, which include Change Data Capture (CDC) and Structured Streaming.

• Create and orchestrate data ingestion and integration workflows using Azure Data Factory (ADF), Auto Loader, and Lakeflow.

• Enforce data governance, security measures, and access control protocols through Unity Catalog.

• Develop REST APIs, integrations, and automation scripts utilizing Python.

• Establish CI/CD, version control, and deployment automation workflows using Git and Databricks Workflows.

⛳️ Requirements

• Advanced expertise in Databricks, PySpark, Spark SQL, and Delta Lake.

• Significant experience with DLT / Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming.

• Practical experience with Liquid Clustering, Auto Loader, and Delta table optimization techniques.

• Proficient in Unity Catalog, governance, security, lineage, and access control implementations.

• Experience in defining and executing data architecture strategies, including the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold).

• Hands-on familiarity with Azure Data Factory (ADF) for orchestration, data ingestion, and integration tasks.

• Experience in developing REST APIs and advanced Python programming for creating integrations, microservices, and automations.

• In-depth knowledge of Spark performance tuning, Photon, partitioning, shuffle operations, and execution plan analysis.

• Familiarity with Databricks Workflows, CI/CD practices, Git, and Declarative Automation Bundles.

• Ability to optimize workloads, scale computational resources, and minimize processing expenses.

• Experience with observability, monitoring, and maintaining production pipelines.

• Capacity to act as a technical reference, mentor team members, and establish best practices in data engineering.

• Certifications: Databricks Certified Data Engineer Associate; Databricks Certified Data Engineer Professional.

🏝️ Benefits

• Meal and/or food allowance

• Health and dental insurance

• Health insurance for your pet

• Home office allowance (for remote positions)

• Welcome kit included in onboarding

• Life insurance

• Zenklub, providing psychological and nutritional support

• TotalPass

• Certification and Continuous Learning Program with partners (Adobe, Anthropic, AWS, Google, Microsoft, Salesforce)

• Partnerships and discounts to enhance your professional growth

• Access to UOL content

• Maternity kit

• Extended maternity and paternity leave

• Employee referral program