Senior Efficacy Research and Psychometrics Analyst

This is a fully remote position, open to applicants in United States, +2 more locations.

📋 Description

• Oversee the technical aspects of psychometrics and efficacy research for assessments and learning materials at Khan Academy Kids.

• Act as the main technical liaison for external partners involved in psychometric research.

• Assist in designing studies, sharing data, conducting psychometric analyses, and interpreting results.

• Extract, cleanse, and securely distribute assessment data to research partners on a regular basis.

• Analyze item difficulty, score distributions, growth metrics, concurrent validity, and implement IRT-based scoring.

• Convert research insights into product enhancements, including scoring models and rules for discontinuing assessments.

• Create and publish technical documentation, starting with the technical manual for the PreK assessment.

• Recruit and onboard schools and district sites for conducting psychometric and efficacy studies.

• Execute Memoranda of Understanding (MOUs), align on consent procedures and participant incentives, and organize site visits for direct child assessment data collection.

• Collaborate with legal teams to draft and negotiate data-sharing agreements.

• Coordinate Khan Academy Kids’ involvement in ongoing efficacy studies and contribute to the design of future studies.

• Develop and maintain the Khan Academy Kids KPI dashboard in collaboration with engineering, including integrations with App Store Connect and Google Play Console.

• Prepare monthly and quarterly business reviews, set periodic targets, and conduct regular app performance reporting.

• Manage ad hoc SQL data requests for targeted outreach to districts and grant reporting.

• Work alongside product managers, engineers, designers, academic experts, and external research organizations.

⛳️ Requirements

• An advanced degree (PhD or Master's) in psychometrics, educational measurement, quantitative methods, or a related discipline — or equivalent applied experience.

• A solid understanding of psychometric methods, including item analysis, evidence of reliability and validity, growth modeling, and Item Response Theory (IRT).

• Proficiency in SQL and a statistical programming language (e.g., R or Python) for data extraction, cleaning, analysis, and visualization.

• Experience with (or willingness to learn) business intelligence tools like Looker, and comfort in building dashboards and data pipelines.

• Knowledge of responsible data-sharing practices in educational research, including considerations for student data privacy (e.g., FERPA, COPPA) and experience with pseudonymized and deidentified data.

• Outstanding communication skills, capable of translating complex psychometric findings into clear implications for product teams, funders, and school partners.

• Strong project management and organizational capabilities, with the ability to juggle multiple external partners, recurring deliverables, and data-sharing timelines with meticulous attention to detail.

• Demonstrated cross-cultural competency, showing self-awareness, awareness of others, and the capability to adopt inclusive perspectives, attitudes, and behaviors to foster inclusion and belonging within the organization.

• Driven by Khan Academy’s mission “to provide a free, world-class education for anyone, anywhere.”

• Experience with (or a keen interest in) early childhood education and assessment of young learners is advantageous.

🏝️ Benefits

• Competitive salaries.

• Generous paid time off as necessary – Your well-being is a priority.

• Eight pre-scheduled Wellness Days in 2026, occurring on a Monday or Friday for an extended weekend boost.

• Remote-first culture that accommodates your time zone, with flexibility as required.

• Generous parental leave.

• An exceptional team that trusts you and grants you the freedom to excel.

• The opportunity to utilize your skills towards a meaningful mission and work on impactful products that are shaping the future of education.

• Opportunities to connect through affinity, ally, and social groups.

• 401(k) with 4% matching.

• Comprehensive insurance coverage, including medical, dental, vision, and life.