Subject-Matter Experts – AI Benchmark Question Writers, Reviewers

This is a fully remote position, open to applicants in Thailand.

📋 Description

• Develop original, exam-style questions in the specified language and designated subject area.

• Create both short-form questions and long-form connected reasoning tasks evaluated against a defined rubric.

• Assess each question using five AI models: GPT, Gemini, Claude Opus, Kimi, and Muse Spark.

• Ensure questions are accepted only when at least three out of five models provide incorrect answers.

• Generate a minimum of five non-challenge questions per domain that are answered correctly by the models.

• Include the correct answer, reasoning, difficulty explanation, and model errors for each question, translated into English.

• Document all work using a structured Google Sheets template.

• Evaluate accepted questions based on criteria such as form, ambiguity, multi-step reasoning, correctness, and reasoning explanation.

• Provide comments on any issues and either approve or return questions as necessary.

⛳️ Requirements

• Must be a native speaker of Thai, Filipino, Vietnamese, Korean, or Russian.

• Hold a university degree in one of the specified domains (Master's or PhD preferred) and/or possess professional, teaching, or research experience in the area.

• Exhibit strong written English skills to accurately translate questions, answers, and explanations.

• Be comfortable utilizing AI chat tools and Google Sheets.

• Display precision, patience, and the ability to work independently within deadlines.

• Experience in exam or test writing is advantageous.

• University teaching experience is a plus.

• Expertise in multiple domains is beneficial.

• Previous work in AI training data, evaluation, or annotation is desirable.

• Must be able to complete all tasks by November 2, 2026.

• For each subject area covered, deliver 20 accepted questions (10 short-form and 10 long-form) along with at least 5 non-challenge questions.

🏝️ Benefits

• Question Writers receive payment in USD for each accepted question, with a higher rate for long-form questions.

• Reviewers are compensated per reviewed question or on an hourly basis.

• Flexible working hours are available.

• This position allows for fully remote work.

• Opportunity to contribute to the development and evaluation of AI benchmarks.

• Work may encompass one or multiple subject areas.

• Candidates can start as soon as possible following onboarding.