Senior Data Engineer

This is a fully remote position, open to applicants in Peru.

📋 Description

• Design, construct, and maintain large-scale data processing systems that gather data from various structured and unstructured sources.

• Develop and enhance ELT pipelines utilizing AI-assisted tools to pinpoint bottlenecks, propose optimizations, and automate routine pipeline maintenance tasks.

• Recognize, design, and implement internal process enhancements: leverage AI to automate manual workflows, improve data delivery, and re-engineer infrastructure for increased scalability.

• Establish the necessary infrastructure for optimal extraction, transformation, and loading of data from diverse sources.

• Prepare data for exploration and discovery by data scientists using AI-driven data profiling and quality assessment tools.

• Conduct data wrangling and munging for subsequent analytics and machine learning applications.

• Facilitate large-scale machine learning by designing and maintaining annotated datasets, employing elastic search methods, and creating scalable data lake architectures that support AI/ML workloads.

• Develop and sustain analytics pipelines that produce data and insights to inform business decision-making.

• Collaborate with data scientists, analysts, and business stakeholders to define requirements for dimensional modeling and ETL pipelines.

• Create and uphold data quality frameworks; utilize AI to automate anomaly detection, schema validation, and enforcement of data contracts across pipelines.

⛳️ Requirements

• A minimum of 4 years of experience in a Data Engineer position.

• A graduate degree in Computer Science, Statistics, Informatics, Information Systems, or a related quantitative field.

• Advanced knowledge of SQL and experience with relational databases and query formulation.

• Proven, hands-on experience with AI tools to expedite data engineering tasks, including pipeline development, data quality automation, code generation, or root cause analysis, with specific examples available for discussion.

• Experience in building and optimizing data pipelines, architectures, and datasets.

• Strong analytical skills when handling unstructured and disparate datasets.

• Familiarity with big data technologies such as Hadoop, Spark, Kafka, etc.

• Experience with both relational and NoSQL databases, including Postgres and Cassandra.

• Proficiency in pipeline and workflow management tools like Airflow, Luigi, Azkaban, or similar.

• Experience with AWS cloud services such as EC2, EMR, RDS, and Redshift.

• Familiarity with stream-processing systems like Storm, Spark Streaming, or similar.

• Working knowledge of message queuing, stream processing, and highly scalable data storage solutions.

• Proficiency in object-oriented or scripting languages, such as Python, Java, Scala, C++, or similar.

• Experience supporting cross-functional teams in fast-paced, agile environments.

• High proficiency in English.

🏝️ Benefits

• Health insurance

• Opportunities for professional development