Data Architect, Databricks

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Develop, optimize, and sustain scalable data pipelines along with ETL/ELT processes for analytics and business intelligence.

• Design and implement data architectures, which encompass data lakes, warehouses, and integration layers.

• Establish data quality checks, validation processes, and monitoring systems.

• Create and manage integrations among data sources, third-party tools, and analytics platforms.

• Work closely with Data & Analytics Managers and business stakeholders to grasp requirements and deliver technical solutions.

• Monitor and enhance data pipeline performance, query efficiency, and storage expenses.

• Generate and upkeep technical documentation for data architectures, pipelines, and processes.

• Apply security best practices and ensure adherence to data privacy regulations.

• Utilize AI tools and automation to improve data engineering workflows and productivity.

⛳️ Requirements

• Practical experience with Databricks migration using the databricks/migrate tool, covering notebooks, jobs, clusters, secrets, and ACLs.

• Experience migrating Unity Catalog to Glue Catalog.

• Proficient in notebook and job migration, including notebooks, DAG pipelines, and DLT pipelines.

• Refactor storage paths from /mnt/ and DBFS references to S3-backed locations.

• Execute data migration from Delta Lake format to Iceberg format on S3, which includes Delta tables, raw files, and incremental sync.

• Implement Auto Loader / DLT checkpoint re-bootstrap and event_id deduplication.

• Conduct validation through row counts, checksums, sample queries, and ETL runtime comparison.

• Recreate secret scopes in AWS Secrets Manager.

• Familiarity with AWS S3, Glue, Secrets Manager, IAM fundamentals, Athena, Lake Formation, and EMR.

• Proficient in Spark SQL, PySpark, Java, Python, and Scala.

• Experience in Agile pod delivery, including daily ticket updates, standups, and code reviews.

• Databricks Certified Consultant certification and experience in DBR to EMR on AWS migration are advantageous.

🏝️ Benefits

• Premium Healthcare.

• Meal voucher.

• Maternity and Parental leaves.

• Mobile services subsidy.

• Sick pay and Life insurance.

• CI&T University.

• Colombian Holidays.

• Paid Vacations.

• And many others.

• Diverse, inclusive, and safe work environment.