Data Pipeline Engineer – Microsoft Fabric

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Set up and validate the connection to SharePoint Online as per the specified pilot scope.

• Create pipeline components to ingest chosen documents and their metadata into the OneLake Bronze layer.

• Implement patterns for incremental loading, change detection, retries, execution logging, and exception management.

• Develop Silver layer logic for data cleansing, normalization, deduplication, version control, and conflict resolution.

• Build Gold layer components to prepare verified information products in line with business guidelines.

• Assist in attribute extraction, information enhancement, and validation by business stakeholders.

• Maintain lineage documentation between source documents, pipeline executions, transformed records, and validated outcomes.

• Conduct tests for document creation, updates, versioning, duplication, deletion, exceptions, and modifications.

• Implement operational dashboards, alerts, execution history, logs, and diagnostic support.

• Collaborate with SharePoint and information architecture teams to comprehend source structures, metadata, permissions, and change behaviors.

• Document technical limitations, unsupported functionalities, workarounds, performance, and production considerations.

• Package code, notebooks, configurations, and pipeline definitions using approved version control and release methodologies.

• Generate tests, technical documentation, deployment instructions, and operational runbooks.

• Construct and validate the ingestion flow for documents, metadata, and selected data from SharePoint Online into OneLake.

• Work under the direction of the Fabric and data architecture teams, collaborating with SharePoint, information architecture, AI, and business teams.

⛳️ Requirements

• Practical experience with Microsoft Fabric, OneLake, lakehouses, notebooks, and data pipelines.

• Familiarity with Spark-, PySpark-, or SQL-based transformations.

• Background in pipeline architecture and development, incremental loads, and change detection.

• Understanding of schema management, data quality, observability, recovery, and exception handling.

• Experience in integrating REST APIs, files, document metadata, or Microsoft 365 data sources.

• Proficiency in SQL and at least one data engineering language, such as Python or PySpark.

• Knowledge of medallion architecture and data organization across the Bronze, Silver, and Gold layers.

• Experience with source control, parameterization, environment configuration, deployment, and testing.

• Ability to create repeatable tests that showcase the platform's actual performance.

• Experience in producing technical documentation, runbooks, deployment processes, and support materials.

• Intermediate to advanced English (B2 or higher), with the capability to communicate both verbally and in writing in a corporate and technical setting.

• Availability to work in the Eastern Standard Time (EST) zone.

• Strong analytical thinking and technical investigation abilities.

• Rigor in data validation, test execution, and documentation of results.

• Capability to troubleshoot and investigate integration or processing issues.

• Organization, attention to detail, and engineering discipline.

• Collaboration skills with architects, developers, and business professionals.

• Clear communication regarding platform limitations, risks, failures, and behaviors.

• Proactive approach with an emphasis on automation and continuous improvement.

• Commitment to quality, traceability, observability, and sustainability.

🏝️ Benefits

• CLT employment option.

• SulAmerica health insurance for yourself (nationwide coverage, ward accommodation, and copayments).

• TotalPass gym membership.

• Latest-generation laptop.

• Transportation allowance.

• Meal allowance: BRL 770.00 (based on an average of 22 business days per month), with the option to allocate the amount to VR or a Flash card.

• Candidate referral bonus.

• Creditas financial services partnership.

• Annual performance review with an IDP (Individual Development Plan).

• Training through iblue Academy.

• Udemy training.

• Certifications (AWS, Microsoft, IBM, and H2O).

• Educational partnerships (with potential financial support tied to your performance review).

• Structured Y-shaped career path (you can choose between a management or specialist track).

• Benefit packages provided by the cooperative, which you can select based on your needs.

• 15 days of paid leave after the 12th month.