Senior Data Developer, Azure

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Pipeline Development: Design and implement ELT/ETL pipelines for the ingestion, transformation, and delivery of data from various sources, such as spreadsheets, SharePoint, and relational databases, adhering to architectural standards;

• Data Platform Construction: Create and structure the layers of the Data Lake/Lakehouse (Bronze, Silver, and Gold / Medallion Architecture) utilizing optimized formats like Delta Lake and Parquet;

• Transformation and Modeling: Craft data transformations using PySpark and advanced SQL, applying architecture-defined models within the platform's analytical layers;

• Ingestion Strategies: Execute loading strategies (full load or incremental) that are suitable for the volume and importance of each data domain—prioritizing efficiency for small-data contexts;

• Data Quality: Establish automated data quality tests and checks within pipelines, ensuring consistency, integrity, and traceability across all layers;

• Cataloging and Governance: Assist in the cataloging and documentation of data assets with a focus on lineage and classification, adhering to governance guidelines set by the Architect and creating a unified data catalog as a core component of platform management;

• Observability: Guarantee the observability of pipelines through logs, alerts, and proactive monitoring;

• Documentation: Record models, transformations, and business rules applied in pipelines, ensuring traceability and maintainability of the overall solution.

⛳️ Requirements

• Demonstrated experience in Data Engineering within cloud environments;

• Extensive experience with PySpark for the development of pipelines and data transformations on distributed platforms;

• Proficient in advanced SQL for queries, transformations, and analytical modeling;

• Familiarity with Microsoft Azure (Azure Data Factory, ADLS Gen2, and/or Microsoft Fabric);

• Experience with data formats such as Delta Lake, Parquet, or ORC;

• Knowledge of Data Lake, Lakehouse, and Medallion Architecture;

• Expertise in data cataloging and lineage, including Unity Catalog (Databricks);

• Preferred / Nice to Have:

• Experience with Databricks and/or Snowflake as a data processing platform;

• Knowledge of Microsoft Fabric (Lakehouses, Notebooks, Pipelines, or Dataflows);

• Familiarity with Microsoft Purview for data cataloging and lineage;

• Understanding of dbt for transformations and data quality testing;

• Experience orchestrating pipelines with Apache Airflow or Prefect;

• DP-700 certification or equivalent;

• Adherence to development best practices: version control with Git, automated testing, documentation, and CI/CD;

• Awareness of Gen AI and autonomous agents applied to data.

🏝️ Benefits

• Health and dental insurance;

• Meal and food allowance;

• Childcare assistance;

• Extended parental leave;

• Partnerships with gyms and health and wellness professionals via Wellhub (Gympass) / TotalPass;

• Profit Sharing (PLR);

• Life insurance;

• Continuous learning platform (CI&T University);

• Discount club;

• Free online platform dedicated to promoting physical and mental health and wellbeing;

• Expectant parent and responsible parenthood course;

• Partnerships with online course platforms;

• Language learning platform;

• And many others.