Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop batch and streaming data ingestion pipelines

• Design and organize data lakes and data warehouses

• Generate optimized datasets for machine learning applications

• Implement embedding pipelines

• Construct vector indexing for Retrieval-Augmented Generation (RAG)

• Ensure the integrity, governance, and security of data

• Optimize costs related to storage and processing

• Collaborate with AI Engineers in structuring feature stores

⛳️ Requirements

• Proficiency in programming language: Python

• Advanced SQL skills

• Knowledge of Scala (optional)

• Experience with Apache Airflow

• Familiarity with dbt

• Proficient in Prefect

• Experience with Spark

• Knowledge of Pandas

• Proficient in PySpark

• Experience with data lakes (S3, GCS, Azure Blob)

• Familiarity with data warehouses (BigQuery, Snowflake, Redshift)

• Knowledge of NoSQL databases

• Experience with vector databases (Pinecone, Weaviate, FAISS)

• Familiarity with Kafka

• Knowledge of Pub/Sub

• Experience with Docker

• Familiarity with Kubernetes

• Knowledge of cloud platforms (AWS, GCP, or Azure)

• Skills in data modeling

• Experience with ETL / ELT processes

• Understanding of distributed processing

• Ability to design scalable data architectures

• Experience with unstructured data (text, logs, PDFs)

• Familiarity with DataOps concepts

• Knowledge of data versioning and quality

🏝️ Benefits

• Meal and/or food allowance of R$950.00 per month on an iFood card

• SulAmérica health insurance (fully covered by the company)

• MetLife dental plan

• Pet health insurance

• Life insurance

• TotalPass and Wellhub memberships to promote physical activity anytime, anywhere

• Birthday day off to celebrate your special day as you choose

• Partnership with Sesc

• Collaboration with OnHappy (discounts on hotels and flights)

• Partnerships with educational institutions

• Childcare reimbursement

• Provision of work equipment

• A culture fostering feedback and professional development

• A dynamic environment with ample growth opportunities

• Payroll-deducted loans with some of the lowest interest rates available

• Variable compensation to acknowledge achievement, performance, and cultural alignment