Data Architect, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Construct, enhance, and oversee scalable data pipelines as well as ETL/ELT processes.

• Design and implement data architectures that encompass data lakes, warehouses, and integration layers.

• Establish data quality validations, checks, and monitoring systems.

• Develop and sustain integrations between data sources, third-party applications, and analytics platforms.

• Work closely with Data & Analytics Managers and business stakeholders to comprehend requirements and provide technical solutions.

• Track and enhance pipeline performance, query effectiveness, and storage expenses.

• Generate and maintain technical documentation for data architectures, pipelines, and processes.

• Apply security best practices and ensure adherence to data privacy regulations.

• Utilize AI tools and automation to improve data engineering workflows and productivity.

⛳️ Requirements

• Practical experience with Databricks migration using the databricks/migrate tool, covering notebooks, jobs, clusters, secrets, and ACLs.

• Experience in migrating from Unity Catalog to Glue Catalog.

• Proficiency in notebook and job migration, including notebooks, DAG pipelines, and DLT pipelines.

• Refactoring storage paths from /mnt/ and DBFS references to S3-backed locations.

• Migrating data from Delta Lake format to Iceberg format on S3, including Delta tables, raw files, and incremental sync.

• Experience with streaming using Auto Loader and DLT checkpoint re-bootstrap.

• Event ID deduplication skills.

• Validation expertise using row counts, checksums, sample queries, and ETL runtime comparison.

• Capability to recreate secret scopes in AWS Secrets Manager.

• Familiarity with AWS S3, Glue, Secrets Manager, IAM basics, Athena, Lake Formation, and EMR.

• Proficient in Spark SQL, PySpark, Java, Python, and Scala.

• Agile pod delivery experience, including daily ticket updates, standups, and code reviews.

• Databricks Certified Consultant certification (Associate/Professional) is preferred.

• Preferred experience in migrating from DBR to EMR on AWS.

🏝️ Benefits

• Health and dental insurance.

• Meal and food allowance.

• Childcare assistance.

• Extended paternity leave.

• Partnerships with gyms and health and wellness professionals through Wellhub (Gympass) TotalPass.

• Profit Sharing and Results Participation (PLR).

• Life insurance.

• Access to a continuous learning platform (CI&T University).

• Discount club.

• Free online platform dedicated to physical, mental, and overall well-being.

• Pregnancy and responsible parenting course.

• Collaborations with online learning platforms.

• Language learning platform.

• Health and Well-being team, inclusion specialists, and affinity groups.

• Support and accommodations for individuals with disabilities.