Senior Data Engineer – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, maintain, and enhance data pipelines (ETL/ELT) utilizing Databricks

• Create data processing solutions leveraging Apache Spark and PySpark

• Handle large data volumes, ensuring optimal performance, scalability, and reliability

• Develop and oversee data ingestion, transformation, and availability processes

• Contribute to the design and advancement of Data Lake and Lakehouse architectures

• Create and refine queries using SQL

• Apply best practices for data quality, governance, and security

• Conduct performance analysis and optimization of jobs, pipelines, and queries

• Assist in technical and architectural decisions regarding the data platform

• Establish monitoring and failure-handling processes for pipelines

• Engage with code versioning and CI/CD workflows

• Participate in agile ceremonies and collaborate with Data Engineering, Analytics, Architecture, and business teams

• Offer technical support and mentorship to team members, sharing best practices and enhancing solutions

⛳️ Requirements

• Extensive experience as a Data Engineer, preferably at a Senior level

• Demonstrated experience with Databricks

• In-depth expertise in Apache Spark and PySpark

• Advanced proficiency in SQL

• Knowledge of Delta Lake

• Experience in building and maintaining ETL/ELT pipelines

• Familiarity with Data Lake, Lakehouse, and/or Data Warehouse architectures

• Experience in processing large data sets

• Understanding of data modeling and transformation

• Experience with Git and CI/CD methodologies

• Knowledge of data security, governance, and quality best practices

• Capable of analyzing performance and troubleshooting pipelines

• Analytical and collaborative mindset with the ability to independently lead technical tasks

• Preferred: experience with Databricks Unity Catalog

• Preferred: knowledge of Databricks Workflows/Jobs

• Preferred: familiarity with Medallion Architecture (Bronze, Silver, Gold)

• Preferred: knowledge of Python for Data Engineering

• Preferred: experience with data services on AWS, Azure, or GCP

• Preferred: knowledge of Apache Airflow or Azure Data Factory

• Preferred: experience with Terraform

• Preferred: certifications related to Databricks or cloud platforms

• Preferred: experience in high-volume environments and mission-critical data solutions

🏝️ Benefits

• Meal and/or food allowance

• Partnership with Sesi and Sesc, granting access to health, wellness, and leisure services

• Collaborations with educational institutions offering exclusive discounts on courses and programs

• Opportunities for career advancement and involvement in strategic projects

• Chance to work at a rapidly growing company within the market