Data Engineer – Mid-Level

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Create and develop ETL/ELT pipelines in Databricks or through open-source methods, facilitating data ingestion and processing within a medallion architecture on Delta Lake.

• Standardize and ingest PDF documents and images alongside their metadata.

• Conduct data curation and anonymization in line with Brazil’s LGPD, generating datasets suitable for RAG, few-shot learning, and testing purposes.

• Construct and manage vector indexing, encompassing chunking, embedding generation, incremental index updates, and version control.

• Prepare versioned datasets (golden sets) for accuracy assessment and benchmarking tasks.

• Establish feedback-loop persistence and contribute to the Quality and SLA Dashboard.

• Connect databases with inference services and APIs for other systems, considering performance, cost, monitoring, and alerting requirements.

• Streamline processes through automation utilizing Workflows/Jobs, CI/CD, and data quality evaluations.

⛳️ Requirements

• Proficient in Python and SQL at an advanced level.

• Knowledgeable in data modeling and medallion architecture / Lakehouse.

• Experience with unstructured data pipelines (PDFs, images, and text) and preparing data for RAG.

• Familiar with vector databases and embeddings (Databricks Vector Search, pgvector, or equivalent).

• Skilled in data quality, testing, and version control.

• Practical experience with Databricks: Delta Lake, Workflows/Jobs, notebooks, PySpark, and Spark SQL.

• Knowledge of Git and CI/CD practices.

• Familiarity with cloud environments (Azure, AWS, or GCP).

• Understanding of LGPD and protocols for handling sensitive data.

• Experience with Unity Catalog in a governed setting (preferred).

• Knowledge of MLflow, Databricks Model Serving, and Mosaic AI (preferred).

• Familiarity with Delta Live Tables / Lakeflow and Auto Loader (preferred).

• Background in the insurance or financial services sector (preferred).

• Experience with Terraform or Databricks Asset Bundles (preferred).

• Databricks Data Engineer Associate or Professional certification (preferred).

🏝️ Benefits

• Anywhere Office: geographic flexibility to work from the location that makes the most sense for you, depending on the project.

• Discount partnerships.

• Employee referral program with rewards.

• Birthday day off.

• TotalPass/Gympass.

• Health and wellness initiatives.

• Opportunities to relax at our offices, featuring beanbags, a pool table, video games, a lounge, a fully equipped kitchen, and coffee.