Lead Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, develop, and advance a petabyte-scale AWS data platform.

• Create and enhance scalable data pipelines utilizing JVM languages, Spark, Python, and cloud-native services.

• Serve as a principal-level technical leader by guiding engineers, performing code reviews, and advocating for best practices.

• Participate in architectural and design discussions with architects, product owners, and engineering leaders.

• Plan and execute complex technical projects within agile value-stream teams.

• Ensure operational readiness through established coding standards, effective testing practices, robust monitoring strategies, and thorough release procedures.

• Assist with production deployments and uphold platform stability.

• Investigate and implement GraphQL integrations, vector databases, and AI/LLM-based approaches.

⛳️ Requirements

• Expert-level experience in software and data engineering for large-scale data platforms.

• Extensive experience in building petabyte-scale systems using Java-based languages like Scala and Apache Spark.

• Profound knowledge of the AWS data ecosystem, including Glue, S3, Athena, Managed Airflow, and Iceberg.

• Advanced comprehension of distributed systems and highly parallelized workloads.

• Strong abilities in query optimization, data partitioning, and efficient storage methodologies.

• Mastery of the AWS cloud ecosystem; familiarity with Azure/GCP is a plus.

• Experience in shaping long-term architecture and platform design.

• Exceptional skills in code review, mentorship, and technical leadership.

• Practical experience in agile environments and collaborating with various engineering teams.

• Proficient in version control and multi-repository collaboration using Git, GitHub, and Bitbucket.

• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience.

• Relevant professional experience.

• Proficient in advanced English.

• Willingness to travel to São Carlos/SP as needed.

• Experience with DBT or contemporary transformation frameworks.

• Knowledge of concurrent and parallel programming.

• Familiarity with Python, Angular, and TypeScript.

• Experience with vector databases such as pgvector and Redis vector fields.

• Practical application of AI/LLM techniques in data platforms, data access, or software quality.

🏝️ Benefits

• Inclusive recruitment initiatives.

• Professional development opportunities.

• Affinity groups that support underrepresented communities: ExperianPride, Ubuntu, Women in Experian, Aspire, and Connecting Generations.