Natural Language Measurement Specialist

This is a fully remote position, open to applicants in United States.

📋 Description

• Assist in defining and enhancing the measurement of language-based performance in critical educational environments.

• Convert measurement concepts into feedback and annotation systems powered by LLM.

• Design and implement studies that validate, ensure fairness, and confirm reliability.

• Act as the measurement expert for cross-functional colleagues—psychometricians, engineers, and data scientists.

• Utilize psychometric principles—reliability, validity, dimensionality, and measurement invariance—in LLM-based feedback and annotation systems.

• Create and employ strategies for identifying and reducing bias in language-based scoring.

• Establish and recommend guidelines for annotation, criteria for feedback quality, and standards for acceptable model performance.

• Contribute to the design, fine-tuning, and assessment of LLM-based feedback and annotation systems.

• Work alongside senior team members to convert measurement insights into operational systems.

• Lead and participate in research studies that assess model performance and reinforce assessment validity.

⛳️ Requirements

• A Master’s or PhD (or nearing completion) in a quantitative discipline such as Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Data Science, or a related field (measurement-oriented training is highly valued).

• A strong grounding in measurement theory, covering reliability, validity, and fairness; knowledge of IRT, generalizability theory, or similar frameworks is preferred.

• Experience in analyzing language or text data, with knowledge of NLP or large language models.

• Proficiency in Python programming and familiarity with data science libraries (e.g., pandas, NumPy, PyTorch).

• Proven capability to conduct thorough research (e.g., thesis, publications, or applied research projects).

• Strong background in statistical analysis and experimental design.

• Experience working with both structured and unstructured data.

• Meticulous attention to detail and a dedication to producing high-quality, reproducible work.

• Willingness to travel 5-10 times annually to College Board offices or for College Board-related business.

• A curious mindset and a desire to learn new tools, methods, and areas.

• Considerate of the implications of AI systems, particularly regarding fairness and validity.

• Capable of articulating technical and measurement concepts clearly to varied audiences.

• Comfortable collaborating in a cross-functional team environment.

🏝️ Benefits

• Annual bonuses and potential for merit-based raises and promotions.

• A mission-driven workplace where your contributions are significant.

• A team that prioritizes your growth and success.