Senior Data Engineering Analyst

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Engage in the assessment and diagnosis of existing processes in Databricks that utilize legacy frameworks based on Hive Metastore.

• Conduct a comprehensive mapping of utilized and unused tables, identifying technical impacts and opportunities for enhancement.

• Collaborate with the Data Engineering team to define migration strategies, data asset reorganization, or discontinuation.

• Develop new data ingestion and transformation processes for modern architectural layers, including bronze, silver, and gold.

• Adjust existing pipelines for the consumption of new tables and governed structures.

• Execute processes for cleaning and permanently deleting obsolete or unused tables.

• Ensure best practices in versioning, deployment automation, and continuous integration.

• Participate in agile ceremonies and collaborate with multidisciplinary squads using the Scrum methodology.

• Contribute to the advancement of governance, quality, and standardization within the analytical environment.

⛳️ Requirements

• Strong experience as a Data Engineer in modern data environments.

• Proficiency in Databricks, with hands-on experience in notebooks, workflows, and environment administration.

• Advanced knowledge of Python for data engineering and process automation.

• Experience with Apache Spark for distributed processing and handling large data volumes.

• Familiarity with CI/CD, version control, and automated pipeline deployment.

• Understanding of Unity Catalog and data governance concepts within the Databricks ecosystem.

• Experience with layered architecture (Bronze / Silver / Gold).

• Solid understanding of SQL and data modeling.

• Experience working in agile teams utilizing Scrum.

• **Differentials**

• Experience in migration projects from Hive Metastore to Unity Catalog.

• Knowledge of performance optimization in Databricks.

• Experience with cloud environments (AWS, Azure, or GCP).

• Understanding of DataOps and pipeline observability.

🏝️ Benefits

• Flex Card for Meals: Use the Flex Card to eat wherever you want, without worries.

• Comprehensive Health Coverage: Enjoy full health and dental coverage without headaches.

• Well-being with Wellhub: Access gyms, studios, and physical activities to maintain body and mind balance.

• Support for the Little Ones: Count on Childcare Assistance to help with your child's care.

• Extended Parental Leave: More time to enjoy and care for your family during important moments.

• Comfort for Remote Workers: Receive Home Office Assistance to support additional expenses.

• Gift on Your Day: Get a Gift Voucher on your birthday to celebrate in your preferred way!

• Commute Stress-Free: The Transportation Voucher will take you to and from work easily and hassle-free.

• Grow with Top Partnerships: Take advantage of our partnerships with giants like Google, Microsoft, BSTQ, Amazon, and more to boost your career with valuable certifications.