Senior Systems Software Engineer – Accelerated Spark Applications

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Design, construct, and manage a hosted Spark service control plane on Kubernetes, which includes the Java REST API, Spark Connect runner, Kyuubi, along with deployment utilizing Helm charts and Flux.

• Take ownership of code reviews for one or more services from start to finish and ensure they remain releasable on the release schedule.

• Enhance the cuDF Spark runtime and CPU/GPU workload router using Scala and Java, ensuring interoperability with Delta Lake, Apache Iceberg, and Unity Catalog.

• Investigate production failures on clusters handling over a thousand concurrent jobs in collaboration with customers and upstream library teams, driving resolutions to completion.

• Implement metrics, dashboards, and alerts; engage in the on-call rotation and maintain runbooks.

• Offer insights and recommendations on infrastructure, continuous integration, and testing strategy decisions.

⛳️ Requirements

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related field (or equivalent experience).

• Over 9 years of experience in software development.

• Strong foundational knowledge in distributed systems.

• Proficient in Java or Scala, along with C++ or Python.

• Practical experience in operating services on Kubernetes, whether on-premises or in the cloud.

• Experience in owning or reviewing a long-lived server, which includes REST front ends, proxies or request routers, JDBC or Thrift services.

• Experience in supporting enterprise or internal production customers, including on-call duties.

• Familiarity with using AI tools in software development.

🏝️ Benefits

• Equity

• Comprehensive benefits package