Senior Software Engineer, Data Product

This is a fully remote position, open to applicants in Hawaii, +6 more states.

📋 Description

• Take charge of backend services that provide EDD predictions to merchants and internal users, focusing on APIs, caching, contracts, and ensuring reliability under production loads.

• Develop Python services tailored for high-throughput and low-latency workloads.

• Spearhead API design, service decomposition, and conduct cross-team technical reviews for data product surfaces that include rules automation, ML-based recommendations, analytics, and configuration systems.

• Ensure reliability and observability for the services you create, incorporating instrumentation, alerting, runbooks, and incident response.

• Collaborate with data science teams to transition model outputs into production, managing the API layer, serving infrastructure, and operational reliability of ML-driven features.

• Construct and sustain feature pipelines that connect offline training with online inference, prioritizing consistency and data quality.

• Lay the groundwork for MLOps within the team, including model deployment patterns, versioning, rollback procedures, A/B test infrastructure, and experiment tracking integrations.

• Equip ML systems for observability—monitoring latency, throughput, drift signals, and prediction quality—so that issues are identified before they affect merchants.

• Assess frameworks, tools, and architectural patterns for ML serving, making practical recommendations based on production experience.

• Set the technical direction for backend and ML systems within the Data Products team, advocating for and leading architectural decisions that balance development speed with long-term maintainability.

• Conduct design reviews, elevate standards in code reviews, and establish engineering practices for the team to adhere to.

• Provide mentorship to other engineers in Software or ML engineering.

• Utilize AI tools in your own workflow and share insights with the team.

⛳️ Requirements

• Over 8 years of experience in building production backend systems, with a significant portion dedicated to ML-powered features.

• Proficient in Python backend development, specifically with FastAPI (or a similar async framework), strong knowledge of PostgreSQL (including schema design, query optimization, and migrations), and practical experience with event-driven architectures such as Kafka.

• Proven ability to oversee distributed systems throughout their entire lifecycle: design, launch, monitoring, and iteration.

• Practical experience in deploying and operating ML models as APIs, not just training them.

• Familiarity with ML lifecycle tools (such as MLflow or equivalent) and the discipline of treating models as production artifacts with appropriate tracking, registry, and promotion processes.

• Comfortable with concepts of model versioning, shadow modes, canary deployments, A/B tests, and rollback strategies, including understanding when each method is applicable.

• Capable of instrumenting an ML system to capture relevant signals (latency, throughput, drift, prediction quality) and articulate issues to a non-ML audience when discrepancies arise.

• Committed to writing high-quality, maintainable code and managing problems from design through to long-term production behavior, holding that standard during design and code reviews.

• Able to clearly communicate trade-offs, including difficult decisions such as "we shouldn't ship this yet" or "the bottleneck isn't the model."

• Collaborates effectively with Data Science, viewing ML as a collective responsibility rather than segregating operations from data science roles.

🏝️ Benefits

• Flexible work arrangements

• Professional development opportunities