Senior Software Engineer, Backend – Workload Safety

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish and enhance safety standards for Affirm's EKS fleet workloads.

• Take full ownership of standards, encompassing everything from research and prototyping to onboarding, self-service guidance, automation, and policy guardrails.

• Spearhead cross-team initiatives from design to fleet-wide implementation.

• Provide mentorship to engineers while ensuring adherence to standards for design, code review, and operational readiness.

• Classify workloads based on criticality and resource profile.

• Troubleshoot and resolve issues related to stuck workloads and node lifecycle.

• Identify safe operating limits for clusters and formulate control plane scaling strategies.

• Examine workload resource profiles to inform EC2 instance type strategy and automate this analysis.

• Assess the impact of standards on availability, latency, risk, and cost to prioritize the development roadmap.

• Participate in on-call support for Kubernetes runtime challenges.

• Work collaboratively with service, batch, ML workload, deployment, capacity, and SRE teams.

• Monitor Kubernetes community best practices and tailor them to fit Affirm's requirements.

⛳️ Requirements

• Over 5 years of experience managing Kubernetes in a large-scale production environment, preferably with Amazon EKS across multiple clusters.

• Profound knowledge of Kubernetes resource management and scheduling, including requests and limits, OOM and CPU throttling behaviors, affinity and anti-affinity, taints and tolerations, as well as topology spread constraints.

• Strong understanding of pod lifecycle and disruption management, covering graceful termination, pod disruption budgets, and node draining.

• Experience in enforcing configuration standards through admission control and policy-as-code tools like OPA Gatekeeper or Kyverno.

• Proficient in Go or Python, specifically for production infrastructure software such as Kubernetes controllers, operators, or admission webhooks.

• Excellent production debugging capabilities with a focus on transforming recurring issues into systemic solutions.

• Proven history of leading technical initiatives across various teams, promoting the adoption of shared standards, and mentoring engineers.

• Experience in measuring the impact of infrastructure changes on availability, latency, and cost.

• Strong written communication skills.

• A Bachelor’s degree in a relevant field or equivalent practical experience.

🏝️ Benefits

• Equity rewards.

• Monthly stipends for health, wellness, and technology expenses.

• Comprehensive medical coverage for employees and their dependents at no cost.

• Dental and vision insurance for employees and their dependents.

• Flexible time off policy.

• Generous holiday schedule.

• Employee stock purchase plan (ESPP) with discounted Affirm stock.

• Remote-first work flexibility.

• In-person onboarding experience.

• Inclusive interview process with accommodations for candidates with disabilities.