Senior Data Engineer, Data Lake

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, test, and maintain data ingestion, transformation, and quality-check pipelines utilizing PySpark/SparkSQL on Databricks.

• Develop and adjust Airflow DAGs (MWAA) for orchestrating pipelines.

• Craft and enhance SQL queries for data transformation and validation purposes.

• Implement and oversee data quality checks using Great Expectations or similar tools.

• Engage in code review processes.

• Troubleshoot pipeline failures and data quality challenges with support from senior engineers.

• Create and uphold documentation for datasets and pipelines.

• Take part in sprint planning, estimation, and retrospective sessions.

• Adhere to team-defined CI/CD, testing, and governance protocols.

• Contribute to the reliability and integrity of Visa's corporate data lake.

⛳️ Requirements

• Must reside in Brazil.

• Proficient in English at a B2 level or higher (Upper-Intermediate).

• A minimum of 3 years of relevant work experience with a Bachelor's or Associate's Degree, or 5+ years of applicable work experience.

• Experience with Python for automation and data processing tasks.

• Understanding of data pipeline concepts (batch, ETL/ELT patterns).

• Basic knowledge of Apache Spark (PySpark or SparkSQL).

• Familiarity with Databricks (jobs, workflows, cluster management, tuning).

• Intermediate to advanced SQL skills (joins, window functions, CTEs).

• Knowledge of Amazon S3 data lake design (partitioning, layout, lifecycle management).

• Basic understanding of cloud concepts (AWS: S3, IAM, CloudWatch).

• Experience in data modeling (dimensional/Kimball, medallion layers).

• Proficient in Git/GitHub workflow and code review practices.

• Preferred: Experience with Delta Lake, Apache Airflow/MWAA, CDC patterns, data quality principles, basic Terraform knowledge, and BI integration (Superset, dashboarding).

🏝️ Benefits

• Flexible remote work arrangement.

• Opportunity to make a significant impact at scale.

• Growth of skills and professional development through meaningful projects.

• Scheduled notice for potential Visa office presence may be available for remote roles.