Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Assist in enhancing the data engineering aspects of the platform, ensuring the processing environments are available, high-performing, scalable, and reliable.

• Oversee processes, address incidents, implement ongoing improvements, and develop new functionalities.

• Collaborate with architecture, development, and business teams to maintain data quality and governance.

• Monitor and ensure the availability of data processing environments.

• Identify, analyze, and resolve incidents occurring in production environments.

• Implement enhancements to optimize the performance, scalability, and reliability of data pipelines.

• Investigate inconsistencies, validate data quality, and apply corrective and preventive measures.

• Design and maintain processing pipelines utilizing Azure Databricks (PySpark and Spark SQL).

• Create and manage ingestion and orchestration workflows with Azure Data Factory.

• Document technical solutions, processes, workflows, and operational procedures.

• Contribute to the advancement of architecture and best practices within data engineering.

⛳️ Requirements

• Proficient in Microsoft Azure.

• Experienced with Azure Databricks (PySpark and Spark SQL).

• Familiarity with Azure Data Factory (ADF).

• Knowledge of Azure Synapse Analytics.

• Strong proficiency in Python.

• Experience with PySpark.

• Robust SQL skills (Spark SQL and T-SQL).

• Background in developing and supporting data pipelines.

• Experience in monitoring and resolving incidents in production settings.

• Understanding of data quality, integrity, and governance principles.

• Familiarity with best practices in cloud data architecture.

• Experience in optimizing the performance of data pipelines.

• Knowledge of distributed processing using Apache Spark.

• Proficient in code versioning (Git/Azure DevOps).

• Understanding of Data Lake, Data Warehouse, and ETL/ELT concepts.

• Experience in monitoring and ensuring observability of data environments.

• Familiarity with agile methodologies (Scrum/Kanban).

• Experience working in critical production environments.

• Bachelor's degree in Computer Science, Computer Engineering, Information Systems, Software Engineering, Systems Analysis and Development, or related fields.

• A postgraduate degree or certifications in Data Engineering, Cloud Computing, or Microsoft Azure is a plus.

🏝️ Benefits

• Company-subsidized Health Insurance for employees.

• Option to add dependents to the Health Insurance with payroll deduction.

• Optional Dental care.

• Option to include dependents in Dental Care with payroll deduction.

• Meal allowance or Food voucher.

• Transportation voucher (optional).

• Impact & Care - a personal guidance program providing confidential emotional support and counseling in psychological, legal, financial, social, and pet-related areas at no cost for employees and their legal dependents.

• Gympass - Wellhub (Access to over 700 gyms across Brazil with plans starting at R$ 29.90 deducted through payroll).

• Option to include dependents in Gympass - Wellhub (up to 3 dependents - paid via credit card).

• Access to Udemy through our intranet.

• Partnerships with major consumer brands.

• Agreement with SESC for employees and their dependents.

• Discounts with educational institutions (undergraduate and postgraduate) and language/certification schools.

• Group life insurance.