Senior Data Engineer – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Implement and enhance the Databricks data platform

• Develop and improve data pipelines utilizing Spark/PySpark

• Create ingestion, transformation, and processing solutions for substantial data volumes

• Collaborate with Delta Lake and resources within the Databricks ecosystem

• Contribute to the design and modeling of scalable data solutions

• Apply data engineering best practices related to quality, governance, and observability

• Identify and address performance issues in data pipelines and processes

• Optimize processing, storage, and query performance

• Work with Unity Catalog and associated data governance and security capabilities

• Engage in defining and executing pipeline orchestration and integration strategies

• Partner with technology and business teams to determine optimal technical solutions

• Ensure the quality, reliability, scalability, and maintainability of the developed solutions

• Contribute to the advancement of the project's data engineering standards and best practices

⛳️ Requirements

• Solid, hands-on experience with Databricks

• Advanced proficiency with Apache Spark and PySpark

• Experience with Delta Lake

• Knowledge of and experience with Unity Catalog

• Background in implementing and enhancing data pipelines

• Familiarity with data architecture and data engineering best practices

• Experience in optimizing and troubleshooting performance in Spark/Databricks

• Proven experience in processing large volumes of data

• Ability to work independently when analyzing issues and defining technical solutions

• Experience in corporate or enterprise environments

• Preferred: Databricks certification

• Preferred: Experience with AWS

• Preferred: Familiarity with Terraform and Infrastructure as Code (IaC) practices

• Preferred: Knowledge of or experience with Kafka and other streaming technologies

• Preferred: Experience with data quality

• Preferred: Understanding of observability tools and practices

• Preferred: Previous involvement in large-scale projects

• Preferred: Experience in high-volume environments with high-availability requirements

🏝️ Benefits

• Collaborative environment

• Learning and career growth opportunities

• Excellent workplace

• Inspiring colleagues and leaders

• Freedom to propose and develop innovative projects

• Autonomy and ownership

• Opportunities to develop your talents

• An environment where you feel valued