Data Engineer – Databricks, DBT, Streaming, ML, AI

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Establish architectural frameworks, standards, and guidelines for the AI-focused data platform.

• Create and develop mechanisms for the ingestion, processing, and provisioning of unstructured data, including PDFs, audio, images, and videos.

• Specify data structures suitable for utilization by AI/ML and generative AI models.

• Automate, enhance, and recommend modifications to current data processes.

• Conduct proofs of concept and practical assessments to validate solutions prior to scaling.

• Act as a technical authority within an enablement team, providing support to multidisciplinary teams.

• Construct and refine scalable data pipelines that integrate with external applications.

⛳️ Requirements

• Extensive experience with Databricks and Apache Spark for large-scale distributed processing.

• Proficient in Python and SQL as they pertain to data engineering (knowledge of Scala is a bonus).

• Experience in processing unstructured data, such as documents/PDFs, audio, images, or video, for AI applications.

• Proven track record in defining data architecture and providing technical guidance (beyond just execution).

• Familiarity with data preparation techniques for AI/ML and generative AI, including feature engineering, chunking, embeddings, and model consumption.

• Experience in integrating data with external systems through APIs, SaaS services, and streaming.

• Background in cloud computing, ideally with GCP.

• Knowledge of Delta Lake, including Lakehouse architecture, data versioning, and governance.

• Experience in data modeling and transformation using DBT.

• Practical experience in developing scalable and robust data pipelines (ETL/ELT).

• Familiarity with code versioning systems (Git) and CI/CD practices.

• Experience working in agile environments and collaborating with diverse teams.

• Hands-on approach with the initiative to propose, test, and enhance processes.

🏝️ Benefits

• Porto Seguro health insurance.

• Porto Seguro dental plan.

• Profit Sharing (PLR).

• Childcare assistance.

• Alelo meal and grocery vouchers.

• Home office allowance.

• Partnerships with educational institutions.

• Support for certifications, including cloud certifications.

• Livelo points.

• TotalPass.

• Mindself.