Technical Product Manager – Pedagogy, AI Quality

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish quality benchmarks and objectives for exceptional reading and writing supports, informed by research in literacy, special education, and instruction for multilingual learners.

• Develop and sustain datasets, evaluation rubrics, and assessment tools.

• Perform consistent error analysis and trace review utilizing human labeling, LLM-as-judge assessments, and rule-based evaluations.

• Create, test, and enhance prompts.

• Conduct model comparisons and regression analyses.

• Analyze educator feedback, support ticket information, product analytics, and engage in user research with students and educators.

• Maintain a prioritized list of AI quality and scaffold user experience challenges.

• Draft precise product specifications.

• Collaborate with engineering teams to scope, develop, and validate solutions, product updates, and new features.

• Take ownership of the quality and usability of Medley’s scaffold suite over time.

• Prioritize product enhancements and new features to support educators and enhance student outcomes.

⛳️ Requirements

• Expertise in pedagogy, particularly in scaffolding literacy instruction for struggling readers, special education, and/or multilingual education.

• Ideally possess experience in classroom teaching, intervention, instructional coaching, or curriculum design.

• Practical experience with AI evaluations, including human labeling, LLM-as-judge rubric creation, defining AI metrics targets, and analyzing errors in LLM outputs.

• Proficient in AI products; daily engagement with LLMs and experience in designing complex AI systems for products.

• Experience in defining new features and collaborating with engineers to implement them.

• Ability to effectively communicate challenges and solutions, navigating technical trade-offs and user experience questions.

• Demonstrated initiative, sound judgment, and strong writing skills.

• A graduate degree and/or significant hands-on experience in special education, reading/literacy, ESL/ELD, or bilingual education is a plus.

• Familiarity with AI evaluation and observability tools such as LangSmith, Braintrust, Arize, Langfuse, or annotation platforms is beneficial.

• Proficiency in a language other than English is advantageous.

• Previous experience at an edtech company, or in content quality, assessment design, or psychometrics is a plus.

• Understanding of accessibility standards such as WCAG and assistive technology is preferred.

• Must address the work authorization sponsorship question for the United States.

🏝️ Benefits

• Equity.

• Collaborate directly with founders and engineering leadership.

• Engage in meaningful technical and design ownership.

• Opportunity to influence a product that enhances access to learning for students across the nation.