Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, create, and sustain data infrastructure and architecture for the Artificial Intelligence initiatives within the Legal department.

• Develop, build, and maintain robust, scalable, and efficient data pipelines.

• Ingest, extract, transform, cleanse, standardize, and enhance large volumes of data.

• Structure and implement Lakehouse and Medallion architectures (Bronze, Silver, and Gold).

• Convert raw data into structured and dependable data products, emphasizing the Gold layer.

• Prepare and provide data for Artificial Intelligence, Machine Learning, and LLM solutions.

• Create ETL/ELT processes for the integration of structured and unstructured data.

• Establish data validation, quality, traceability, and monitoring processes.

• Organize and prepare legal, litigation, contractual, and document data.

• Develop solutions for managing large volumes of documents and unstructured information.

• Collaborate with Data Scientists, AI professionals, Technology teams, and Legal experts.

• Ensure security, governance, access control, and compliance with Brazil’s LGPD data protection regulations.

• Identify opportunities to enhance performance, reduce costs, and improve efficiency across the data infrastructure.

⛳️ Requirements

• Strong experience in Data Engineering and data solution architecture.

• Proficiency in SQL and Python for data manipulation, transformation, and processing.

• Experience in developing ETL/ELT pipelines.

• Familiarity with Lakehouse architecture and the Medallion model (Bronze, Silver, and Gold).

• Experience with Apache Spark and PySpark.

• Knowledge of cloud-based data storage and processing platforms.

• Experience in integrating and processing both structured and unstructured data.

• Understanding of data modeling and the construction of Data Lakes and Data Warehouses.

• Proficient with pipeline orchestration tools such as Apache Airflow or similar technologies.

• Knowledge of data governance, quality, lineage, and security.

• Familiarity with preparing data for Machine Learning models and Large Language Models (LLMs).

• Experience with Generative Artificial Intelligence projects and LLM-based solutions (a plus).

• Understanding of Retrieval-Augmented Generation (RAG) architectures (a plus).

• Experience with document processing, OCR, and extracting and structuring textual information (a plus).

• Knowledge of vector databases and embedding techniques (a plus).

• Experience working with legal, litigation, contractual, or banking-sector data (a plus).

• Familiarity with data and AI services on AWS, Azure, or Google Cloud (a plus).

• Experience with DataOps and CI/CD practices applied to Data Engineering (a plus).

🏝️ Benefits

• This position is also open to individuals with disabilities (PwD).