Master Data Developer, Databricks

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design, optimize, and sustain scalable data pipelines and ETL/ELT processes for analytics and business intelligence.

• Establish data architectures, including data lakes, warehouses, and integration layers.

• Execute data quality checks, validation processes, and monitoring systems.

• Develop and uphold integrations between data sources, third-party tools, and analytics platforms.

• Collaborate with Data & Analytics Managers and business stakeholders to comprehend requirements and deliver technical solutions.

• Oversee and enhance data pipeline performance, query efficiency, and storage costs.

• Generate and maintain technical documentation for data architectures, pipelines, and processes.

• Apply security best practices and ensure adherence to data privacy regulations.

• Utilize AI tools and automation to improve data engineering workflows and productivity.

⛳️ Requirements

• Practical experience with Databricks migration using the databricks/migrate tool, encompassing notebooks, jobs, clusters, secrets, and ACLs.

• Experience in migrating Unity Catalog to Glue Catalog.

• Expertise in notebook and job migration, inclusive of DAG pipelines and DLT pipelines.

• Refactor storage paths from /mnt/ and DBFS references to S3-backed locations.

• Migrate data from Delta Lake format to Iceberg format on S3, which includes Delta tables, raw files, and incremental synchronization.

• Experience with streaming using Auto Loader / DLT checkpoint re-bootstrap and event_id deduplication.

• Conduct validation using row counts, checksums, sample queries, and ETL runtime comparisons.

• Recreate secret scopes in AWS Secrets Manager.

• Familiarity with AWS services: S3, Glue, Secrets Manager, IAM basics, Athena, Lake Formation, and EMR.

• Proficiency in programming and query languages: Spark SQL, PySpark, Java, Python, and Scala.

• Engage in agile pod delivery, including daily ticket updates, standups, and code reviews.

🏝️ Benefits

• Premium Healthcare.

• Meal voucher.

• Maternity and Parental leaves.

• Mobile services subsidy.

• Sick pay-Life insurance.

• CI&T University.

• Colombian Holidays.

• Paid Vacations.