Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, develop, and enhance contemporary data solutions within the Azure Cloud framework, utilizing Databricks as the primary analytics processing tool.

• Create, maintain, and optimize data ingestion, transformation, and delivery pipelines through Azure Databricks.

• Implement scalable ETL/ELT processes to handle large volumes of both structured and unstructured data.

• Develop solutions employing Python, PySpark, and SQL.

• Construct and sustain Lakehouse architectures utilizing Azure Data Lake Storage (ADLS Gen2) and Databricks.

• Apply data engineering best practices, including observability, monitoring, and governance.

• Automate data processes using Git, CI/CD, and Infrastructure as Code methodologies.

• Ensure the quality, integrity, traceability, and security of data.

• Support initiatives in Data Science, Machine Learning, Analytics, and Power BI.

• Collaborate with architects, business analysts, and technology teams to create solutions that align with business objectives.

• Identify opportunities for enhancing platform performance and reducing costs through FinOps strategies.

• Engage in defining standards, frameworks, and best practices in data engineering.

• Mentor junior staff members.

⛳️ Requirements

• Significant experience as a Senior Data Engineer in corporate settings.

• Demonstrated expertise with Microsoft Azure.

• Advanced proficiency with Azure Databricks.

• Familiarity with Data Lake and Lakehouse architectures.

• Experience in Analytics, Big Data, and Modern Data Platform initiatives.

• Hands-on, solution-focused mindset.

• Ability to operate independently in complex scenarios.

• Strong communication abilities with both technical and business teams.

• Analytical thinking and results-driven approach.

• Strong dedication to ownership and quality standards.

• Capability to mentor junior team members.

• Comfort working in agile and collaborative settings.

• Advanced knowledge of Python.

• Advanced skills in PySpark.

• Advanced expertise in SQL.

• Familiarity with Azure Data Lake Storage Gen2.

• Experience with Azure Data Factory.

• Proficiency in Git and code version control.

• Knowledge of DevOps and CI/CD principles for data.

• Experience in analytical and dimensional data modeling.

• Ability to optimize Spark pipeline performance.

• Understanding of data governance, quality, and security principles.

• Experience in data integration via APIs, relational databases, and enterprise systems.

• Bachelor’s degree in Computer Science, Engineering, Information Systems, Statistics, or a related field.

• Familiarity with Delta Lake.

• Knowledge of Unity Catalog.

• Experience with Databricks Workflows and Lakeflow.

• Proficiency in streaming with Spark Structured Streaming.

• Familiarity with Power BI.

• Experience with Microsoft Fabric.

• Knowledge of Event Hub or Kafka.

• Experience with Terraform or Bicep.

• Familiarity with Machine Learning and MLOps solutions.

• Understanding of SAP Datasphere, SAP BDC, or SAP environments.

• Experience with FinOps practices in Azure.

• Relevant certifications such as DP-203, Azure Data Engineer Associate, or Databricks Data Engineer.

🏝️ Benefits

• Flexible working hours

• Support for children with disabilities

• Variable Compensation Program

• Private pension plan

• Length-of-service bonus

• Online therapy and nutritional counseling

• Newborn benefit package

• Corporate university

• Eyeglasses and contact lens reimbursement

• Medication allowance

• Vaccination allowance

• Gympass

• Meal and food allowances

• Transportation allowance

• Health and dental insurance

• Life insurance