Tech Lead

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Comprehend the product, its audience, and data sources, providing independent technical leadership for requirements.

• Develop and sustain PySpark and SQL pipelines within Databricks.

• Collaborate with medallion architecture, Delta Lake, Unity Catalog, and Databricks Workflows.

• Prepare data for generative AI services, which includes embeddings, vector indexes, quality assurance, and versioning.

• Integrate data with AI services and Databricks Genie.

• Consume and expose data through APIs, ensuring security and observability.

• Design the gold layer and make data accessible to the product, Power BI, and analytical applications.

• Create reusable frameworks and implement quality validations.

• Employ version control, CI/CD, testing, and code review methodologies.

• Work in short cycles alongside the product team, focusing on incremental delivery.

• Document pipelines and technical decisions thoroughly.

• Assist with technical estimates, guide the squad, and advocate for best practices.

• Ensure classification, anonymization, and access control in collaboration with Data Governance.

⛳️ Requirements

• Extensive experience in data engineering, serving as the technical reference for the squad.

• Proficiency in Databricks, PySpark, and Spark SQL within production settings.

• Familiarity with medallion architecture: landing, bronze, silver, and gold layers.

• Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.

• Experience in data engineering for generative AI, including data preparation, embeddings, vector indexes, quality assurance, and versioning.

• Expertise in data integration through APIs and services, ensuring security and observability.

• Skills in analytical data modeling in the silver and gold layers.

• Proficient in Git, CI/CD, code review, testing, and quality validations.

• Experience in agile product teams with a focus on incremental delivery.

• Strong communication abilities with both technical and business stakeholders.

• Degree in IT or a related field.

• Knowledge of Databricks Genie and enterprise standards for generative AI applications.

• Understanding of data governance as applied to AI and Brazil’s LGPD, encompassing classification, anonymization, lineage, and access control.

• Experience with Oracle and replication via GoldenGate/CDC.

• Proficient in data extraction from SAP via Open Hub.

• Familiarity with SQL Server and Microsoft Graph (SharePoint/M365).

• Experience with Power BI.

• Knowledge of credential management using service identities and secrets vaults.

• Ability to create technical documentation in Confluence.

🏝️ Benefits

• Meal and/or food allowance.

• Health and dental insurance.

• Health insurance coverage for your pet.

• Home office allowance for remote positions.

• Welcome kit provided during onboarding.

• Life insurance coverage.

• Access to Zenklub for psychological and nutritional support.

• TotalPass subscription.

• Certification and Continuous Learning Program in partnership with organizations like Adobe, Anthropic, AWS, Google, Microsoft, and Salesforce.

• Partnerships and discounts to foster your professional development.

• Access to UOL content.

• Maternity kit provided.

• Extended maternity and paternity leave.

• Employee referral program.