Engenheiro de Dados PL, Focado em IA

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Engage in the development of data ingestion, transformation, and enrichment pipelines for AI utilization.

• Work with both structured and unstructured data (text, PDFs, HTML, audio, among others).

• Implement chunking, embeddings, and vector indexing processes.

• Build and maintain datasets aimed at the corporate knowledge matrix.

• Develop pipelines using Databricks (Spark / PySpark).

• Operate within a medallion architecture (bronze, silver, and gold).

• Integrate data with vector databases (Azure AI Search, pgvector, etc.).

• Ensure the performance, scalability, and reliability of the pipelines.

• Apply best practices in data quality (completeness, consistency, and versioning).

• Implement policies for data updating, retention, and purging.

• Guarantee traceability and auditability of the data utilized by the models.

• Collaborate with AI/ML teams in data preparation and optimization.

• Support information retrieval strategies (RAG).

• Optimize data to enhance the relevance and accuracy of model responses.

⛳️ Requirements

• Solid experience in data engineering.

• Proficiency in Python and/or PySpark.

• Experience with Databricks and Spark (batch and/or streaming).

• Experience with data pipelines (ETL/ELT).

• Data modeling (Data Lake / Lakehouse).

• Experience with unstructured data (documents, texts, etc.).

• Integration and consumption of APIs.

• Ability to work autonomously in building pipelines.

• Knowledge of modern data architecture.

• Experience with data processing and preparation for AI.

• Experience in complex environments with multiple integrations.

• Hold one of the following certifications: Microsoft DevOps Engineer Expert; AWS Developer; Google Cloud Architect; Azure Developer Associate; IBM Cloud or variations; or ITIL 4 Foundation.

🏝️ Benefits

• Meal allowance or meal voucher.

• Discounts on courses, universities, and language institutions.

• Stefanini Academy — a platform offering free, updated online courses with certification.

• Mentoring.

• Benefits club for consultations and exams.

• Medical assistance.

• Dental assistance.

• Discounts and advantages at top establishments.

• Travel club.

• Pet care agreement.