Mid-Senior Data Developer, PySpark, AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, implement, and maintain comprehensive data pipelines (ELT/ETL) emphasizing reliability, reprocessing, and cost-effectiveness.

• Manage job orchestration and routines utilizing Airflow (on AWS) and serverless functions (AWS Lambda); assist with deployments on EKS when necessary.

• Structure data in Snowflake (bronze/silver/gold layers) and create transformations in dbt (macros, tests, documentation).

• Develop integrations and services in Python using FastAPI to facilitate data exposure/consumption and automate workflows.

• Ingest, standardize, and version market data, focusing on historical data and distribution (batch; limited streaming).

• Write optimized SQL (including tuning, partitioning/clustering in Snowflake and Postgres).

• Utilize Pandas for on-the-fly processing, validations, and transformation prototyping.

• Guarantee quality (dbt tests, data checks), observability (logs/metrics/alerts), security (access controls, masking), and thorough documentation.

• Collaborate with business, analytics, and product teams to establish SLAs, data contracts, and governance standards.

⛳️ Requirements

• Strong expertise in SQL (query optimization, relational and analytical modeling).

• Practical experience with Snowflake (warehousing, roles, tasks, performance, cost management) and dbt (models, tests, sources, exposures).

• Proven experience in building data pipelines using Airflow (DAGs, sensors, retries, SLAs) and AWS Lambda.

• Familiarity with Postgres (ingestion, basic replication/CDC or maintenance, routines).

• Competence in Python for data handling (Pandas) and API development using FastAPI.

• Experience with batch-oriented ELT/ETL, Git version control, and CI/CD (secure deployment of pipelines/dbt models).

• Understanding of data security and governance (access control, lineage, documentation, sensitive data).

• Proficiency in technical English for reading documentation.

🏝️ Benefits

• Health and dental insurance

• Food and meal allowance

• Childcare assistance

• Extended parental leave

• Partnerships with gyms and health & wellness professionals via Wellhub (Gympass) / TotalPass

• Profit Sharing (PLR)

• Life insurance

• Continuous learning platform (CI&T University)

• Employee discount club

• Free online platform dedicated to physical and mental health and well-being

• Prenatal and responsible-parenting course

• Partnerships with online course platforms

• Language learning platform