Senior Data Tech Lead – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Lead the technical direction of data engineering projects on Databricks within the Azure environment.

• Collaborate with the Data Architect to establish standards and references, eliminate obstacles, and facilitate the creation of scalable and reliable pipelines in line with the Medallion architecture.

• Provide technical guidance to Data Engineering teams through design processes, reviews, and critical decision-making.

• Co-develop the technical vision, coding standards, modeling conventions, and best practices for Lakehouse in partnership with the Data Architect.

• Design and manage Databricks pipelines across the Bronze, Silver, and Gold layers.

• Implement and uphold DataOps methodologies, including Git, CI/CD, unit/integration/data-quality testing, and thorough documentation.

• Enhance job and pipeline performance through partitioning, Spark optimization, cost management, observability, and alerting mechanisms.

• Ensure that security, access control, and data policies within Azure/Databricks comply with governance standards.

• Work closely with business and analytics stakeholders to convert requirements into scalable solutions and achievable delivery plans.

• Mentor engineers, uphold standards, and develop templates and playbooks to expedite team progress.

⛳️ Requirements

• Extensive experience in Data Engineering and relevant projects, with a background as a Tech Lead/Senior in providing technical and operational guidance to teams.

• Significant hands-on expertise in Databricks, including proficiency in PySpark and Spark SQL.

• Familiarity with batch pipelines and a foundational understanding of streaming when necessary.

• Proficient in Delta Lake and practical implementation of the Medallion architecture (Bronze/Silver/Gold).

• Strong experience with Azure data services, encompassing Azure Databricks, Azure Data Lake Storage (ADLS), and integration with security/identity services like Azure Active Directory.

• Excellent SQL and Python programming skills.

• Knowledge of dimensional/consumption modeling fundamentals and best practices regarding contracts/schemas.

• Experience with DataOps, Git, CI/CD for data pipelines, automated testing, and well-organized documentation.

• Demonstrated ability to review design/code, address performance/scale challenges, and guide teams effectively.

• Advanced English language skills for communication with stakeholders and the production of technical documentation.

• Nice to have: Familiarity with Unity Catalog, Databricks Workflows/Repos, Delta Live Tables.

• Nice to have: Experience with Great Expectations/Soda and/or dbt in the Lakehouse.

• Nice to have: Knowledge of Airflow/Databricks Workflows and observability through metrics, logs, and alerts.

• Nice to have: Experience in CDC/streaming ingestion using Event Hubs/Kafka.

• Nice to have: Consulting experience, exposure to international clients, and certifications in Databricks/Azure.

🏝️ Benefits

• Health and dental insurance.

• Meal and food allowance.

• Childcare assistance.

• Extended paternity leave.

• Partnership with gyms and wellness professionals through Wellhub (Gympass) TotalPass.

• Profit Sharing and Results Participation (PLR).

• Life insurance.

• Access to a continuous learning platform (CI&T University).

• Membership in a discount club.

• Free online resources focused on physical, mental, and overall wellness.

• Pregnancy and responsible parenting courses.

• Collaborations with online learning platforms.

• Language learning platform access.

• Support from a Health and Well-being team, inclusion specialists, and affinity groups.

• Accommodations and support for individuals with disabilities during the selection process.