Data Architect, AWS, Databricks

This is a fully remote position, open to applicants anywhere in the world.

📋 Description

• Execute the comprehensive migration of the existing environment, currently hosted on Databricks on Azure, to AWS, which includes developing a new data model and reorganizing legacy pipelines and processes;

• Establish and refine the architecture for the Corporate Data Platform (Lakehouse);

• Ensure compliance with the target model based on AWS and Databricks;

• Create architecture standards, frameworks, and best practices;

• Lead the formulation of the migration strategy, including waves, prioritization, and dependencies;

• Migration and Modernization: Spearhead the modernization of the legacy Data Warehouse (from Azure/DataStage to AWS/Databricks);

• Define migration strategies: Incremental versus Big Bang;

• Maintain operational continuity throughout the transition;

• Governance & Security: Establish and enforce standards for Data governance, Access control, and Data quality and lineage;

• Ensure adherence to corporate policies and compliance with LGPD (Brazilian data protection law);

• DataOps & Standardization: Develop standardized and reusable pipelines;

• Implement best practices for Continuous Integration and Continuous Deployment for data;

• Minimize reliance on manual processes and enhance standardization;

• Integration and Ecosystem: Create integrations with various sources and on-premises systems;

⛳️ Requirements

• Proficient in Cloud and AWS Platform, including S3, Glue, IAM, Lake Formation, CloudWatch, and CloudTrail;

• Familiarity with Databricks: Unity Catalog, Delta Lake, notebooks, clusters, and policies;

• Understanding of modern Lakehouse-based architecture;

• Experience in data modeling (Data Warehouse, Lakehouse – Bronze/Silver/Gold);

• Experienced in data pipelines (ETL/ELT);

• Proficient in advanced SQL and Python, with knowledge of tools like Airflow, Control-M, and distributed orchestration;

• Background with ADF/DataStage (legacy);

• Experience in CI/CD for data (Azure DevOps, Git, pipelines);

• Knowledge of Data Quality, Data Contracts, and Data Lineage;

• Experience with data cataloging and corporate governance;

• Familiarity with security and compliance (LGPD, access control, sensitive data);

• Knowledge of integrating multiple sources: APIs, relational databases, NoSQL, and mainframe;

• Experience in distributed and domain-driven architecture;

• Understanding of migration strategies: Replatform, Refactor, Rewrite;

• Knowledge of monitoring tools (e.g., Datadog, CloudWatch);

• Ability to define SLAs/SLOs;

• Experience troubleshooting critical data pipelines;

🏝️ Benefits

• Competitive salary and performance-based bonuses;

• Comprehensive health and wellness benefits;

• Opportunities for professional development and growth;

• Flexible working hours and remote work options;

• Collaborative and inclusive work environment;