Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, implement, and enhance Data Engineering solutions utilizing the Databricks ecosystem.

• Establish and execute scalable data architectures grounded in the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold).

• Create, optimize, and sustain data pipelines through Databricks, PySpark, Spark SQL, and Delta Lake.

• Implement solutions for data ingestion, processing, and integration, including Change Data Capture (CDC) and Structured Streaming.

• Orchestrate pipelines and integrate systems by using Azure Data Factory (ADF).

• Develop REST APIs, microservices, and automation processes employing Python.

• Ensure adherence to data governance, security protocols, lineage tracking, and access control measures.

• Enhance performance, scalability, and cost efficiency of processing.

• Provide support for observability, monitoring, and operational tasks for data pipelines.

• Act as a technical resource for the team, mentoring colleagues and establishing best practices in Data Engineering.

⛳️ Requirements

• Extensive, advanced experience in delivering production-level projects using Databricks.

• High-level proficiency in PySpark, Spark SQL, Delta Lake, and Apache Spark.

• Familiarity with DLT / Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming.

• Knowledge in Liquid Clustering, Auto Loader, and optimization of Delta tables.

• In-depth understanding of Unity Catalog along with governance, security, lineage, and access control.

• Experience in defining data architectures, particularly the Lakehouse model and Medallion Architecture.

• Proficient with Azure Data Factory (ADF) for orchestration, data ingestion, and integration tasks.

• Expertise in Python for creating REST APIs, microservices, and automation solutions.

• Experience in Spark performance tuning, including Photon, partitioning, shuffle, and execution plan analysis.

• Knowledge of Databricks Workflows, CI/CD, Git, and Declarative Automation Bundles.

• Proven experience in optimizing workloads, sizing compute resources, and minimizing costs.

• Background in observability, monitoring, and providing production support for data pipelines.

• Ability to independently resolve complex technical challenges and make architectural decisions.

• Experience in serving as a technical resource, supporting, and mentoring other professionals.

• Required certification: Databricks Certified Data Engineer Associate.

• Databricks Certified Data Engineer Professional certification is highly preferred.

• Databricks Certified Associate Developer for Apache Spark certification is considered a plus.

• Preferred: Certifications in AWS, Azure, or GCP related to Data Engineering and Solution Architecture.

🏝️ Benefits

• Meal and/or food allowance.

• Health and dental insurance.

• Health insurance for your pet.

• Home office allowance (for remote positions).

• Welcome kit provided during onboarding.

• Life insurance coverage.

• Zenklub membership, offering psychological and nutritional support.

• TotalPass access.

• Certification and Continuous Training Program in collaboration with partners (Adobe, Anthropic, AWS, Google, Microsoft, Salesforce).

• Partnerships and discounts available to promote your professional growth.

• Access to UOL content.

• Maternity kit provided.

• Extended maternity and paternity leave policies.

• Employee referral program.