Data Platform Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Transform Azure Data Factory pipelines into Databricks workflows on AWS utilizing reusable templates.

• Rehost Databricks workspaces on AWS and transition ADLS Gen2 storage to S3.

• Rewrite ADF Web Activities as Lambda functions or Step Functions tasks.

• Substitute ADF-specific scaling with native mechanisms from Databricks.

• Construct and optimize PySpark transformations for production-level data volumes.

• Replace Azure Synapse Serverless with Databricks SQL Warehouse.

• Reconcile the migrated data with source systems.

• Migrate 2,094 ADF pipelines, convert 9,859 pipeline activities, transfer 471 Spark dataflows to Databricks on AWS, and rehost four Databricks workspaces across Development, QA, Pre-production, and Production environments.

⛳️ Requirements

• Proven production experience with Databricks, encompassing workspaces, jobs, and workflows.

• Strong background in Spark and PySpark, particularly with real data volumes and performance tuning.

• Proficiency in production-grade Python.

• Experience in building or migrating Azure Data Factory pipelines, along with a solid understanding of the ADF activity model.

• Familiarity with AWS data services: S3, Glue, Athena, Lambda, and Step Functions.

• Advanced SQL skills, including the ability to read and interpret stored procedures.

• Professional proficiency in written and spoken English.

• Preferred: Experience with Delta Lake, Unity Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka, Databricks certification, data modeling, Great Expectations, SAS/analytics platform integration, and CRM data.

🏝️ Benefits

• 100% remote work.

• Full-time engagement.