Senior Data Engineer – SAP Integration

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Oversee the design, development, and execution of sophisticated Data Engineering solutions, demonstrating strong proficiency in Databricks and data integration from the SAP ecosystem.

• Construct and enhance data pipelines.

• Execute Lakehouse architectures.

• Create ETL/ELT processes.

• Guarantee data quality, scalability, governance, and availability for analytical and strategic contexts.

• Design, develop, and sustain data pipelines using Databricks, ensuring efficiency, scalability, and performance in processing substantial data volumes.

• Formulate data ingestion, transformation, and processing workflows utilizing Apache Spark, PySpark, and SQL.

• Execute Lakehouse and Medallion architectures (Bronze, Silver, and Gold), organizing data flows from ingestion to availability for analytical usage.

• Utilize Delta Lake, Delta Tables, and advanced Databricks features for data storage, versioning, optimization, and management.

• Develop and enhance ETL/ELT processes, integrating various data sources, particularly SAP systems.

• Extract, integrate, and transform data from SAP S/4HANA, SAP ECC, and other solutions within the SAP ecosystem.

• Implement SAP–Databricks integrations through APIs, OData, connectors, and data ingestion methods.

• Create batch and streaming data processing solutions.

• Establish best practices for data governance, quality, security, lineage, and monitoring, including Unity Catalog features.

• Enhance the performance of SQL queries, jobs, and workflows in the Databricks environment.

• Automate and orchestrate pipelines, ensuring reliability and availability of processes.

• Collaborate with Architecture, BI, Analytics, and business teams to develop data-driven solutions.

⛳️ Requirements

• Extensive experience in Data Engineering, focusing on architecture and large-scale data processing initiatives.

• Advanced, hands-on expertise with Databricks, encompassing the development, implementation, and optimization of data pipelines.

• Proficient in Apache Spark, PySpark, and Spark SQL.

• Advanced understanding of Delta Lake, Delta Tables, and Lakehouse architecture.

• Familiarity with Medallion architecture (Bronze, Silver, and Gold).

• Knowledge of Databricks Workflows, Jobs, and orchestration features.

• Experience with Unity Catalog, governance, access control, and data lineage.

• Proficient in SQL and Python as applied to Data Engineering.

• Experience with ETL/ELT processes and the integration of multiple data sources.

• Understanding of the SAP ecosystem, particularly SAP S/4HANA and/or SAP ECC.

• Experience in extracting and integrating SAP data through APIs, OData, RFC, IDoc, or other methods.

• Knowledge of SAP Datasphere, SAP BW/4HANA, or SAP Data Services.

• Experience in cloud environments, preferably AWS, Azure, or Google Cloud.

• Databricks Data Engineer Associate or Professional certification is advantageous.

• Familiarity with Delta Live Tables/Lakeflow Declarative Pipelines is a plus.

• Knowledge of CI/CD, Git, and pipeline deployment automation is a plus.

• Experience with real-time data processing (streaming) is a plus.

• Understanding of SAP BTP and SAP Integration Suite is a plus.

• Experience with modern data architectures and large-scale enterprise environments is a plus.

🏝️ Benefits

• Remote position.

• This position is also open to individuals with disabilities.