Principal Data Scientist

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and implement a measurement strategy for the Connections Platform as well as the wider Data & Infrastructure Platform.

• Assess the platform's impact, outcomes, and business/revenue implications to facilitate executive decision-making.

• Establish meaningful metrics, KPIs, dashboards, and both leading and lagging indicators.

• Create thorough, repeatable evaluations for LLM-based and agentic systems, including LLM-as-a-judge and Braintrust tools.

• Design and analyze A/B tests along with quasi-experimental causal inference studies.

• Utilize switchbacks, difference-in-difference, synthetic control, and interrupted time-series methodologies.

• Set up feedback loops for evaluations that distinguish between activity/adoption and true quality/outcomes.

• Collaborate with engineers to implement events, schemas, pipelines, metrics, and dashboards.

• Communicate complex measurement insights and trade-offs to VP and C-level stakeholders.

• Establish measurement standards for platform teams and assist with Platform-as-a-Product metrics.

• Evaluate multi-dimensional product changes that impact consumers, partners, developers, business, and revenue.

⛳️ Requirements

• Over 10 years of experience in Data Science, UX Research, or Applied Science.

• Significant experience in measuring products where the customer is a developer, internal team, or platform.

• Proven experience in designing evaluations for complex user-facing systems, including LLM-as-a-judge and human review processes.

• Practical knowledge of A/B testing and causal inference methodologies.

• Experience in defining primary metrics, KPIs, OKR metrics, and leading/lagging indicator systems.

• Capability to create event taxonomies, schemas, and instrumentation in collaboration with engineering teams.

• Proficient in Python and SQL.

• Ability to articulate measurement narratives for VP and C-level partners.

• PhD with 4+ years, MS with 6+ years, or BS with 8+ years of experience in a quantitative field (as detailed in preferred qualifications).

• Direct experience assessing complex production systems, including conversational, LLM-based, or agentic products.

• Familiarity with modern evaluation methods and offline/online evaluation frameworks.

• Experience in measuring the impact of AI or agentic tools on productivity, quality, outcomes, and ROI.

• Knowledge of Platform-as-a-Product frameworks.

• Proficient in quasi-experimental and causal inference methods, including difference-in-difference, synthetic control, interrupted time-series, and switchback/interleaved designs.

🏝️ Benefits

• Equity awards based on experience, performance, and location.

• Opportunity for remote work from a location of choice.

• Flexible work arrangements allowing employees to operate from wherever they are most productive through Cloud HQ.

• Opportunities for professional growth and development.

• An inclusive and innovative workplace culture.