Data Engineer – Azure, Databricks, Senior

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and maintain scalable, dependable data pipelines utilizing modern distributed processing technologies to guarantee high-quality and high-performance data ingestion, transformation, and delivery.

• Create data models using Databricks, SAP Datasphere, Data Factory, Azure Data Lake Storage (ADLS), Python, and various databases (SQL Server, Oracle).

• Extract data and establish data lakes and tables.

• Organize RAG for contracts and other documentation.

• Execute ETL processes from CXL APIs and transform/clean the data within Databricks.

• Design and configure data tables in Databricks.

• Develop ETL workflows for analyzing both structured and unstructured data, utilizing APIs for file analysis and data validation.

⛳️ Requirements

• Extensive experience and proficiency with Databricks.

• Familiarity with Azure Data Factory and Azure Data Lake Storage (ADLS).

• Experience in implementing Data Lake and Data Lakehouse architectures.

• Proficiency in Python and Apache Spark.

• Experience with Apache Airflow.

• Knowledge of SQL and NoSQL databases, including PostgreSQL, MongoDB, and Cassandra.

• Understanding of Kafka (preferred).

• Familiarity with dbt (preferred).

• Knowledge of AWS Glue and BigQuery (preferred).

• Awareness of AI / RAG (preferred).

• Understanding of SAP Datasphere (preferred).

🏝️ Benefits

• Comprehensive health insurance plans.

• Flexible working hours and remote work options.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive work environment.