Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, maintain, and enhance large-scale data pipelines in Databricks, primarily utilizing SQL and also PySpark.

• Leverage generative AI, LLMs, and agents to streamline data collection, extraction, structuring, and validation, including from unstructured sources.

• Integrate new internal and external data sources while ensuring traceability and cost-effectiveness.

• Create and maintain APIs that deliver data to the platform’s products.

• Ensure data quality and integrity through automated testing, validation, and monitoring, addressing issues at their core.

• Implement data privacy best practices and ensure adherence to Brazil’s LGPD data protection regulations.

• Enhance processing performance and cost-efficiency in Databricks and AWS.

• Engage in architectural decisions and establish data engineering best practices for the team.

• Utilize Claude Code throughout the development lifecycle and assist the team in maximizing the use of AI tools.

• Share expertise with the team through code reviews and pair programming.

• Collaborate with Product, Data Science, Platform, and business teams to translate requirements into actionable solutions and communicate results effectively.

• Document pipelines, business rules, and data architecture.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent experience.

• Proven experience as a Data Engineer or in a comparable role.

• Advanced SQL proficiency and a solid understanding of data modeling.

• Practical experience with Databricks.

• Hands-on experience with AI applications in data tasks, such as LLMs or agents for collecting, extracting, classifying, or validating information.

• Regular use of AI-driven development tools, including Claude Code, Cursor, GitHub Copilot, or similar technologies.

• Knowledge of Python and PySpark.

• Familiarity with AWS services like S3, Lambda, and EKS.

• Experience with Git and automated testing methodologies.

• Ability to work independently and communicate results effectively.

• Experience in an agile and collaborative work setting.

• Preferred: familiarity with Claude Code.

• Preferred: understanding of Delta Live Tables (DLT) and/or streaming data pipelines.

• Preferred: experience with Unity Catalog.

• Preferred: skills in optimizing Databricks pipeline performance and costs.

• Preferred: experience in building data APIs.

• Preferred: knowledge of CI/CD and best practices for code review.

• Preferred: experience with web scraping and large-scale public data collection.

• Preferred: experience in developing AI agents or integrating with LLM APIs.

• Preferred: knowledge of LGPD as it pertains to data.

• Preferred: proficiency in English for technical reading.

🏝️ Benefits

• Meal and Food Allowance.

• Wellhub/TotalPass membership.

• Home office allowance.

• Health insurance.

• Optional dental insurance.

• Childcare allowance until the child reaches six years old.

• Extended maternity, paternity, and adoption leave.

• Life insurance.

• Birthday Day Off.

• Family Day.

• Mental Health Break: one consecutive week off in December.

• A sense of purpose from contributing to a team that is building something lasting and socially relevant.

• A relaxed, youthful, entrepreneurial, and meritocratic work atmosphere.

• Opportunities for career development and growth.

• An open and inclusive work environment.

• All positions are accessible to individuals with disabilities.