Especialista em Observabilidade de Dados

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop proactive processes to monitor the execution of orchestrated pipelines, ensuring that data is delivered on time and at the agreed frequency with business areas.

• Create and automate data quality tests to identify null, duplicate, or anomalous data before it reaches end users.

• Investigate the root causes of pipeline failures or data anomalies, mitigate issues, and reduce downtime.

• Ensure that data lineage is properly documented and functional.

• Conduct post-mortem meetings following critical incidents, propose architectural improvements, and create playbooks.

• Monitor processing and storage costs, identify inefficient queries and excessive processing use, and recommend optimizations.

• Collaborate with Data Engineers, Analysts, and Data Scientists to establish data contracts and configure observability metrics.

⛳️ Requirements

• Strong knowledge of software engineering fundamentals and best practices.

• Proficiency in Python and PySpark.

• Advanced SQL skills and experience with distributed query engines (Athena, Presto, Trino).

• Practical experience with real-time data processing architectures (Streaming).

• Hands-on knowledge of Spark, Hive, and Presto.

• Familiarity with the Linux environment.

• Robust experience in AWS, focused on the maintenance and evolution of data platforms.

• Experience with AWS IAM and AWS Lake Formation.

• Knowledge of Terraform and automation with GitHub Actions (CI/CD).

• Familiarity with Airflow / MWAA (Amazon Managed Workflows for Apache Airflow).

• Knowledge of AWS Glue (Jobs, Metastore, and Data Quality).

🏝️ Benefits

• Competitive market salary.

• Health insurance.

• Dental plan.

• Private pension.

• Flexible benefits that adapt to your needs.

• Meal assistance.

• Financial counseling.

• Advantage club.

• Home office support.

• Mobility assistance.

• Child support.

• Life insurance.

• Extended leave.

• Promotion of sports, quality of life, and well-being.

• Reduced-rate payroll loans.