Lead Data Engineer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design and manage scalable data pipelines and solutions within Databricks utilizing Unity Catalog and Delta.

• Organize Genie spaces and semantic layers incorporating metadata, certified metric definitions, sample SQL, and reliable assets.

• Oversee and refactor Git/GitHub repositories while creating modular, reusable data components.

• Establish CI/CD workflows to transition Databricks solutions from development to regulated UAT environments.

• Model marketing performance metrics, encompassing email interactions, campaign and promotion histories, offers, packages, and tours.

• Construct point-in-time audience histories and link marketing communications with subsequent sales results.

• Create data solutions at both person and household levels utilizing ID crosswalks and identity-match protocols.

• Enforce suppression, consent, Do Not Email, marketability, and PII-handling regulations.

• Ingest and assimilate data from SQL Server enterprise data warehouses.

• Convert legacy SQL audience logic into governed definitions.

• Deliver audiences from Databricks to Adobe Campaign or Segment with service accounts, encryption, and auditing processes.

• Assess unfamiliar data sources and generate data dictionaries and ERDs.

• Develop data quality tests and validated query/result sets.

• Collaborate with analytics, engineering, marketing, data, and security teams.

⛳️ Requirements

• Over 5 years of experience in Data Engineering or related positions.

• Practical experience in crafting data pipelines using Databricks, including Unity Catalog and Delta.

• Proficient hands-on experience with Python for data processing and transformation tasks.

• Solid SQL skills, with the ability to understand and convert complex existing SQL logic.

• Experience with Git-based development and CI/CD, ideally using Databricks Asset Bundles or similar tools.

• Familiarity with Azure and deploying data solutions across development and regulated UAT environments.

• Experience in modeling data for marketing analytics, customer data, or related business sectors.

• Knowledge of customer identity, person/household data, ID crosswalks, and identity matching protocols.

• Experience in managing consent, suppression, marketability, and PII requirements.

• Experience in integrating data from SQL Server enterprise data warehouses.

• Capability to independently assess unfamiliar data sources and document insights through data dictionaries and ERDs.

• Strong analytical and problem-solving abilities.

• Excellent communication and collaboration skills.

• Proficient in written and spoken English.

🏝️ Benefits

• Certifications in AWS, Databricks, and Snowflake.

• Access to AI learning paths.

• Customized study plans, courses, and additional certifications relevant to the role.

• Availability of Udemy Business courses.

• English language lessons.

• Opportunities for travel to industry conferences and client meetings.

• Career development plans and mentorship programs.

• Special day rewards for birthdays, work anniversaries, and other personal milestones.

• Equipment provided by the company.

• Flexible work options.

• Additional benefits may vary depending on location in LATAM.