Data Engineer – DataOps, AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop, enhance, and manage batch and streaming data pipelines.

• Design and implement data engineering solutions utilizing Databricks and AWS services.

• Employ DataOps methodologies emphasizing process automation, standardization, quality, and reliability.

• Create and maintain CI/CD processes for pipelines, notebooks, jobs, and data components.

• Establish monitoring, observability, logging, and alerting frameworks.

• Ensure data quality, integrity, availability, and traceability.

• Handle the processing and transformation of large datasets.

• Develop solutions using Apache Spark / PySpark and SQL.

• Integrate various data sources and systems.

• Contribute to the definition and advancement of data architecture.

• Support efforts to optimize infrastructure and enhance cloud performance and costs.

• Identify and resolve performance challenges, pipeline failures, and data inconsistencies.

• Collaborate with Data Analysts, Data Scientists, Architects, Engineering teams, and business stakeholders.

• Contribute to the establishment of data engineering standards, best practices, and governance.

⛳️ Requirements

• Solid experience in Data Engineering.

• Proven experience in developing ETL/ELT processes and data pipelines.

• Advanced proficiency in SQL.

• Experience with Python and/or PySpark.

• Understanding of large-scale data architecture and processing.

• Experience with structured and semi-structured data.

• Practical experience with Databricks.

• Knowledge of Apache Spark / PySpark.

• Familiarity with Delta Lake / Delta Tables.

• Understanding of Databricks Jobs, Workflows, and processing mechanisms.

• Knowledge of workload performance best practices and optimization strategies.

• Strong sense of ownership and dedication to delivery.

• Proactive and collaborative approach.

• Ability to tackle challenges and see them through to completion.

• Excellent communication skills and capability to work effectively with multidisciplinary teams.

• Curiosity and a continuous desire for learning and growth.

• Mindset focused on building and ongoing improvement.

🏝️ Benefits

• No benefits, perks, or additional compensation are explicitly stated in the posting.