Senior Data Engineer – AWS, PySpark, Kafka

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and enhance an event-driven architecture on AWS, utilizing Kafka as the core component connecting transactional systems with the analytics platform.

• Create Kafka producers and consumers using PySpark (Spark Structured Streaming), overseeing offsets, checkpointing, idempotency, late data handling, and delivery semantics.

• Establish and promote the design principles for topics, keys, partitioning, retention, compaction, and event contracts.

• Manage schemas within the Schema Registry, ensuring compatibility and evolution without disrupting consumers.

• Implement Change Data Capture (CDC) for transactional databases utilizing Kafka Connect or Debezium.

• Integrate various sources, including custodians, distributors, and market data providers' APIs.

• Construct event persistence in the S3 Data Lake with Iceberg or Delta, incorporating Glue, EMR, and Redshift.

• Enhance performance and cost-effectiveness in Kafka and Spark, focusing on lag, rebalancing, brokers, micro-batches, skew, and spill management.

• Introduce observability and automated quality controls for event flows.

• Ensure data lineage, traceability, access controls, encryption, sensitivity-based segregation, and compliance with Brazil’s LGPD and industry standards.

• Automate provisioning and deployment processes using infrastructure as code and CI/CD methodologies.

• Document architectural decisions (ADRs), engage in code reviews and pair programming, and assist in establishing standards.

• Collaborate with teams in Wealth Management, Sales, Risk, Compliance, and Operations.

• Act as a technical resource for mid-level and junior engineers.

⛳️ Requirements

• Over 6 years of experience in data engineering, with a minimum of 2 years focusing on architectural responsibilities for event-driven platforms.

• Proven experience in launching and maintaining a Kafka streaming platform in production, including incident management and on-call support duties.

• Extensive knowledge of Apache Kafka, including topics, partitions, keys, consumer groups, offsets, retention, compaction, rebalancing, and troubleshooting lag.

• Practical experience with producers and consumers using PySpark (Spark Structured Streaming) integrated with Kafka, covering checkpointing, triggers, watermarks, and state management.

• Strong expertise in the AWS ecosystem: Amazon MSK, S3, Glue, EMR, Lambda, IAM, and Redshift or similar technologies.

• Proficient in Python for data engineering, producing modular, testable, version-controlled code, complete with type annotations and automated testing.

• Advanced SQL skills, including window functions, Common Table Expressions (CTEs), and execution plan analysis.

• Familiarity with Schema Registry and formats like Avro, Protobuf, or JSON Schema, especially concerning schema compatibility and evolution.

• Experience utilizing Kafka Connect or Debezium for Change Data Capture (CDC).

• Knowledge of infrastructure as code tools such as Terraform, CloudFormation, or CDK.

• Proficient in Git, code reviews, and CI/CD processes as applied to data engineering.

• Experience with open table formats like Iceberg or Delta and Lakehouse architecture on S3.

• A strong emphasis on numerical accuracy and delivery guarantees.

• Excellent written communication skills for documenting event architecture.

• Capability to independently investigate issues from start to finish.

• Ability to serve as a technical reference through code reviews, pair programming, and mentorship.

• Skill in translating business needs into sustainable event designs.

• Integrity in estimating scope, risk, and timelines.

• Nice to have: experience in financial markets; regulatory data related to CVM, BACEN, BSM, or ANBIMA; familiarity with Confluent; Kafka Streams; ksqlDB; Apache Flink; Kinesis; Airflow; Step Functions; dbt; data quality, catalog, or observability tools; and AWS or Confluent certifications.

🏝️ Benefits

• Continuous professional development opportunities.

• A dynamic and collaborative workplace environment.

• 100% remote work flexibility.

• The chance to engage in financial markets and investment-related projects.

• Work with leading technologies such as AWS, Kafka, PySpark, and various data engineering tools.

• An inclusive company culture that welcomes applications from individuals with disabilities.