Senior Serverless Spark Migration Engineer

This is a fully remote position, open to applicants in Brazil, +3 more countries.

📋 Description

• Lead the migration of enterprise Spark workloads from on-premise settings to AWS and GCP.

• Evaluate Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.

• Identify migration strategies, including rehosting, replatforming, refactoring, modernization, or retirement.

• Upgrade traditional cluster-based workloads to serverless Spark where applicable.

• Design and implement architectures utilizing AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery.

• Refactor legacy PySpark/Scala/Spark SQL applications for cloud compatibility, scalability, and reliability.

• Migrate workloads using Hadoop, HDFS, YARN, Hive, and on-premise Spark clusters.

• Diagnose and enhance Spark workloads, focusing on partitioning, shuffle behavior, joins, data skew, execution plans, executor configuration, serialization, and SQL execution.

• Conduct performance benchmarking and optimize serverless workloads for efficiency, reliability, and cloud cost management.

• Develop reusable migration tools, automation, templates, and frameworks.

• Implement CI/CD and Infrastructure as Code using tools like Terraform.

• Define testing, validation, cutover, rollback, observability, and production-readiness protocols.

• Collaborate with teams in Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps.

• Oversee the entire migration lifecycle: Discover, Assess, Design, Refactor, Migrate, Validate, Optimize, and Operate.

⛳️ Requirements

• Over 8 years of experience in data engineering, distributed systems, cloud engineering, or platform engineering.

• More than 5 years of hands-on experience with Apache Spark in enterprise environments.

• Strong development experience in PySpark and/or Scala.

• Demonstrated experience in migrating large-scale Spark workloads between different infrastructure platforms.

• Practical experience with both AWS and GCP.

• Familiarity with on-premise Hadoop/Spark ecosystems, including HDFS, YARN, and Hive.

• In-depth understanding of Spark internals and distributed processing.

• Solid foundation in SQL and data engineering principles.

• Experience with cloud data lakes and object storage solutions.

• Strong skills in production troubleshooting and performance optimization.

• Proficiency with CI/CD, Git, and Infrastructure as Code.

• Capability to manage migration projects end-to-end, from discovery and architecture to production cutover and optimization.

• Experience with EMR/EMR Serverless, Dataproc/Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes is advantageous.

🏝️ Benefits

• Flexible remote work arrangement.

• Coverage during Pacific Hours (8:00 AM–5:00 PM PST).