Senior Data Engineer – AI Automations, Azure, AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and build data ingestion pipelines from various external sources (ETLs, APIs, relational and non-relational databases, files, telemetry sensors, etc.);

• Integrate and unify data from diverse sources within the Azure environment, ensuring data quality and consistency;

• Establish reference architecture for data processing and storage;

• Create and oversee data lakes, data warehouses, and data marts within the Azure ecosystem;

• Develop and enhance ETL/ELT processes utilizing Microsoft tools such as Azure Data Factory, Databricks, Synapse Analytics, and Azure Functions;

• Ensure governance, security, and scalability of data pipelines;

• Monitor and optimize data load performance, identify bottlenecks, and suggest enhancements;

• Collaborate closely with data scientists and functional analysts to address business requirements.

⛳️ Requirements

• Proven experience as a Data Engineer, specializing in the creation of scalable pipelines;

• Extensive knowledge of Azure Data Services, including Data Factory, Synapse, Databricks, Azure SQL, Blob Storage, and Monitoring;

• Experience in integrating data through REST/SOAP APIs and processing real-time data streams;

• Familiarity with relational databases (SQL Server, PostgreSQL) and non-relational databases (Cosmos DB, MongoDB);

• Proficient in Python, Spark, SQL, and PowerShell for data manipulation and processing;

• Experience with DataOps, version control (Git), and CI/CD for data pipelines;

• Understanding of data modeling, query optimization, and database tuning;

• Knowledge of data security and compliance, including LGPD and best practices for data storage and sharing;

• Strong ability to document and communicate effectively for interaction with both technical and business teams.

• **Differentials:**

• Experience with event-driven architectures utilizing Azure Event Hub or Kafka;

• Familiarity with machine learning pipelines and MLOps on Azure;

• Microsoft certifications (DP-203 – Azure Data Engineer Associate or equivalent).

🏝️ Benefits

• Work mode: Remote (Home Office);

• Possible travel to client sites (Campinas/SP and others), covered by the company;

• Contract type: PJ (contractor/B2B);

• Team working hours: Business hours / to be specified.