Staff Data Scientist – Experimentation, Causal Inference

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the comprehensive methodology that every team adheres to - hypothesis → metrics → design → power → readout → decision - and make it the standard approach.

• Take charge of the statistical methodology (significance, multiple comparisons, sequential testing, variance reduction such as CUPED) for small-sample, fast-paced scenarios where traditional A/B power is challenging to achieve.

• Develop the methods toolkit for our clustered, hierarchical data (user → sub-account/location → agency), where the units for randomization and analysis vary.

• Implement rigorous causal inference techniques (matching, difference-in-differences, instrumental variables, synthetic control, etc.) when clean experiments are impractical - addressing churn, onboarding, GTM - to distinguish genuine signals from selection bias, seasonality, and mix effects.

• Oversee the design discipline for conducting multiple experiments simultaneously - incorporating layering, orthogonal experiments, holdouts, and safeguards to prevent interference between concurrent tests.

• Collaborate with AI/ML teams to design and assess experiments for AI functionalities, including measurement for non-deterministic, rapidly evolving systems.

• Facilitate the experiment review forum and maintain standards for what qualifies as a legitimate result.

• Create the Experimentation curriculum and templates that enhance the skills of PMs and analysts, ensuring that quality design extends beyond your personal involvement.

• Partner with Analytics Engineering to ensure governed, experiment-ready data and standardized metric definitions.

• Influence leadership and cross-functional collaborators on investment decisions, translating statistical subtleties into clear, actionable guidance.

⛳️ Requirements

• 9+ years of experience in data science, product analytics, or applied statistics, with extensive hands-on involvement in designing and analyzing online controlled experiments at scale.

• Strong foundation in applied statistics - frequentist principles, Bayesian methodologies, power analysis, variance reduction, and understanding the pitfalls of A/B testing (peeking, multiple testing, network/cluster effects).

• Practical expertise in causal inference, demonstrating sound judgment in distinguishing causal results from artifacts of data generation.

• Experience in fast-paced, small-sample, multi-product environments - adept at discerning when a decision requires a clean experiment versus a quick, sufficient analysis.

• Proficient in SQL and have working knowledge of Python or R.

• Ability to influence cross-functional teams and senior leadership - enhancing the quality of experiments without direct authority.

🏝️ Benefits

• EEO Statement: The company is an Equal Opportunity Employer.

• We encourage you to voluntarily provide demographic information for compliance with government recordkeeping, reporting, and other legal obligations.