Staff Software Engineer, Replication Foundations

This is a fully remote position, open to applicants in Washington.

📋 Description

• Establish the technical vision and enhance Temporal’s open-source replication stack from the initial problem definition through to deployment and operational support.

• Direct the design and execution of replication protocols for High Availability namespaces, as well as cross-cluster and cross-region replication, and facilitate migrations between Temporal clusters.

• Propel initiatives focused on scalability and reliability, including multi-cell namespaces, multi-cluster namespaces, load distribution, and handling hotspots.

• Outline and convey system-level guarantees addressing consistency models, ordering, idempotency, failure recovery, performance, and operational behavior.

• Recognize architectural risks and opportunities, shaping the technical roadmap accordingly.

• Collaborate with Cloud Enablement, CGS, Product, and engineering teams.

• Oversee design reviews, enhance implementation and testing methodologies, mentor engineers, and offer technical direction.

• Troubleshoot complex production issues, engage in incident response, and spearhead subsequent improvements.

⛳️ Requirements

• Proven experience in designing and delivering intricate production distributed systems.

• In-depth knowledge of replication, partitioning, consistency, fault tolerance, durability, concurrency, and failure recovery.

• Experience in defining system architecture, protocol behavior, invariants, and associated trade-offs.

• Competence in diagnosing concurrency issues, data inconsistencies, partial failures, and performance bottlenecks.

• Proficiency in writing production-quality concurrent code in Go.

• Familiarity with Java, C++, or other similar systems programming languages is a plus.

• Excellent written and verbal communication skills.

• Capacity to influence technical direction across teams and mentor fellow engineers.

• A thoughtful and inquisitive approach to understanding system behavior under load, during failures, and with varying workloads.

• Experience in designing or maintaining replication protocols or data-plane infrastructure is desirable.

• Familiarity with multi-cluster or multi-region architectures, including active-active or active-passive systems, is an asset.

• Knowledge of database internals, log-based replication, or event-sourced systems is advantageous.

• Previous contributions to large open-source projects or distributed systems infrastructure are a plus.

🏝️ Benefits

• Equity.

• Remote work arrangement.

• AI interview recording is utilized internally for evaluation, with an opt-out option that does not affect candidacy.

• Reasonable accommodations for individuals with disabilities.

• Equal opportunity employment.