Senior Data Engineer – AI, AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and sustain the data infrastructure that underpins the analytical products, ML models, and GenAI solutions for both the company and its clients.

• Create and execute high-scale batch and real-time data ingestion and transformation pipelines.

• Construct and manage lakehouse architectures utilizing AWS S3, Glue, Redshift, and Apache Iceberg.

• Design and orchestrate ML/AI pipelines with AWS SageMaker and Apache Airflow.

• Implement real-time streaming solutions leveraging Apache Kafka and/or AWS Kinesis.

• Investigate and apply GenAI patterns through AWS Bedrock, encompassing RAG pipelines, embedding workflows, and integration with LLMs.

• Apply Data Mesh methodologies to decentralize data domains and enhance team autonomy.

• Ensure data integrity, lineage, and governance through dbt and AWS Glue Data Catalog.

• Enhance cost efficiency and query performance within Redshift and Athena environments.

⛳️ Requirements

• Over 5 years of experience as a Data Engineer with an emphasis on cloud technologies.

• Strong expertise in Python, PySpark, and SQL for extensive data processing and transformation.

• Substantial experience with AWS services: S3, Glue, Redshift, Athena, Lambda, SageMaker, and Kinesis.

• Proficient in orchestrating pipelines using Apache Airflow.

• Familiar with data streaming technologies such as Apache Kafka or AWS Kinesis.

• Knowledgeable in dbt for data transformation and documentation purposes.

• Experienced in Infrastructure as Code with Terraform for the provisioning of data infrastructure.

• Understanding of Apache Iceberg for table management in data lakes.

• Proven interest in AI/ML — experience with ML or GenAI pipelines is highly desirable.

• AWS Certified Data Analytics – Specialty or AWS Certified Machine Learning (preferred).

🏝️ Benefits

• Opportunity for remote work.