Data Engineer, Databricks Lakehouse – Senior

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Collaborate in the development of the Enterprise Lakehouse platform on Databricks, structured into data domains following Data Mesh principles.

• Implement and oversee Unity Catalog as the primary governance layer, facilitating data sharing across domains through Delta Sharing.

• Design data ingestion methodologies (both streaming and batch) and create the medallion architecture template (bronze/silver/gold) for domain applications.

• Construct the platform's data contract framework along with a Data Quality framework.

• Set up CI/CD pipelines for the Databricks platform and automate the provisioning of infrastructure.

• Connect the Databricks platform with the existing Data Mesh product on AWS, ensuring catalog interoperability and compliance with the data contract model.

• Participate in architectural decisions, contribute to technical documentation, and provide mentorship to the team.

⛳️ Requirements

• Databricks: extensive experience with the platform, including workspace administration, cluster management, jobs, and workflows.

• Unity Catalog: expertise in data governance, permission models, lineage, external locations, and storage integration.

• Delta Lake and medallion architecture (bronze, silver, gold) implemented in production settings.

• Data ingestion methodologies: proficiency in streaming and batch processes at scale.

• CI/CD for Databricks (utilizing Databricks Asset Bundles, Repos, and integration with pipelines such as GitHub Actions or Azure DevOps).

• AWS ecosystem knowledge: S3, Glue, Lake Formation, EMR (EC2 and Serverless), Athena, Lambda, IAM, DMS, Kinesis, Step Functions, SNS, SQS, and EventBridge.

• Infrastructure as Code (IaC) experience with Terraform, including the Databricks provider.

• Proficiency in Advanced Python and PySpark for pipeline development and platform automation.

• Advanced SQL skills.

• Familiarity with Data Contracts and metadata tools such as OpenMetadata.

• Experience with Datadog for observability controls and monitoring.

• Desirable:

• Background in Data Mesh projects (domains, data products, federated governance) is a significant advantage.

• Experience in the financial or credit industry is a notable plus.

• Knowledge of observability and monitoring of data platforms (system tables, job metrics, data quality).

• FinOps expertise: monitoring, allocation, and cost optimization in Databricks (DBUs, cluster sizing, compute policies).

• Delta Sharing in scenarios involving cross-company, cross-cloud, or cross-region.

• AWS certifications (e.g., Data Engineer Associate, Solutions Architect) are beneficial.

• Databricks certifications (Data Engineer Associate, Data Engineer Professional, Platform Administrator) are advantageous.

• Understanding of open data contract specifications (Open Data Contract Standard, ODPS).

🏝️ Benefits

• The role is also open to candidates with disabilities (PWD).