Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, develop, maintain, and enhance data pipelines of medium to high complexity.

• Create scalable solutions for the ingestion, transformation, and processing of extensive data volumes.

• Develop applications utilizing Python, SQL, Spark, and PySpark.

• Create, improve, and optimize solutions on the Databricks platform.

• Engage with Data Lake, Lakehouse, and Data Warehouse architectures.

• Define strategies for partitioning, distributed processing, and performance optimization.

• Investigate and address critical issues associated with data processing, integration, and quality.

• Implement mechanisms for observability, monitoring, alerting, and ensuring data quality control.

• Apply best practices related to security, governance, privacy, and data protection.

• Participate in technical decision-making and in establishing development standards and best practices.

• Conduct code reviews, share insights, and offer technical support to Mid-Level and Junior professionals.

• Collaborate with teams from Architecture, Security, Infrastructure, Development, and Business.

• Assist with technical estimates, define scope, and plan deliveries.

• Contribute to the technical and operational enhancement of data solutions.

⛳️ Requirements

• Strong experience in Data Engineering.

• Proven track record with highly complex solutions and large-scale data processing.

• Advanced proficiency in Python.

• Advanced proficiency in SQL.

• Advanced proficiency in Apache Spark/PySpark.

• Experience working with Databricks.

• Familiarity with ETL/ELT processes and distributed processing.

• Knowledge of Azure Data Lake Storage (ADLS).

• Experience with Azure Data Factory or similar orchestration tools.

• Understanding of relational and non-relational databases (SQL and NoSQL).

• Experience with Data Lake, Lakehouse, and Data Warehouse architectures.

• Knowledge of data modeling techniques.

• Experience with Git and CI/CD methodologies.

• Understanding of APIs and system integration.

• Knowledge of security, governance, and data protection principles.

• Familiarity with optimization and tuning of data pipelines and processes.

• Experience in Cloud environments along with development, maintenance, and operational practices for critical business solutions.

• Nice to have: Familiarity with Delta Lake, Databricks Workflows, Unity Catalog, Kafka, Azure Event Hubs, Terraform, Infrastructure as Code (IaC), Microsoft Azure or Databricks certifications, experience in the financial or banking sector, and applied observability frameworks/tools related to data.

🏝️ Benefits

• Meal allowance or meal voucher.

• Discounts on courses, universities, and language schools.

• Stefanini Academy — an online platform providing free, current courses with certificates.

• Mentoring opportunities.

• Benefits club that offers discounts on consultations and medical tests.

• Health insurance coverage.

• Dental insurance coverage.

• Benefits and discounts at top establishments.

• Travel club membership.

• Pet care benefits.