Data Engineer – Mid-Level

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop batch and streaming data ingestion pipelines.

• Architect and organize data lakes and data warehouses.

• Generate optimized datasets for machine learning applications.

• Implement embedding pipelines effectively.

• Construct vector indexing for retrieval-augmented generation (RAG).

• Maintain data quality, governance, and security standards.

• Optimize costs related to storage and processing.

• Collaborate with AI Engineers to design feature stores.

• Establish the infrastructure that supports Machine Learning and Generative AI initiatives.

• Guarantee that data is well-organized, trustworthy, accessible, and scalable for both training and inference.

⛳️ Requirements

• Proficient in programming with Python.

• Advanced knowledge of SQL.

• Scala experience is preferred but not mandatory.

• Familiarity with orchestration and transformation tools like Apache Airflow, dbt, and Prefect.

• Data processing expertise using Spark, Pandas, and PySpark.

• Experience with Data Lakes such as S3, GCS, and Azure Blob.

• Background in Data Warehouses including BigQuery, Snowflake, and Redshift.

• Understanding of NoSQL databases.

• Knowledge of vector databases like Pinecone, Weaviate, and FAISS.

• Experience with streaming technologies such as Kafka and Pub/Sub.

• Proficiency in infrastructure management using Docker and Kubernetes.

• Familiarity with cloud platforms including AWS, GCP, or Azure.

• Skills in data modeling.

• Experience with ETL and ELT processes.

• Knowledge of distributed processing techniques.

• Ability to design scalable data architectures.

• Experience working with unstructured data formats (text, logs, PDFs).

• Understanding of DataOps practices.

• Familiarity with data versioning and quality assurance.

🏝️ Benefits

• Monthly meal and/or food allowance of R$950.00 on an iFood card.

• SulAmérica Medical Plan fully covered by the company.

• MetLife Dental Plan.

• Pet health insurance coverage.

• Life insurance benefits.

• Access to TotalPass and Wellhub memberships to encourage physical activities at any time and place.

• A day off on your birthday.

• Partnership with Sesc.

• Collaborations with OnHappy for discounts on hotels and flights.

• Partnerships with educational institutions.

• Childcare reimbursement provided.

• Provision of work equipment.

• A feedback-driven culture fostering professional growth.

• A dynamic work environment with substantial growth opportunities.

• Payroll-deducted loan options with competitive interest rates.

• Variable compensation based on performance, results, and cultural fit.