AI Evaluation Guidelines & Rubric Specialist

This is a fully remote position, open to applicants in New York.

📋 Description

• Translate program requirements into clear, structured, and actionable instructions for human evaluators.

• Develop guidelines relevant to standard scenarios as well as complex edge cases.

• Define terminology, rating criteria, decision rules, exceptions, and escalation pathways.

• Ensure that instructions are accessible to raters while maintaining domain-specific precision.

• Design detailed scoring rubrics for evaluating generative AI outputs.

• Establish measurable criteria for correctness, relevance, reasoning quality, completeness, and adherence to instructions.

• Create examples and counterexamples that illustrate varying performance levels.

• Align evaluation frameworks with program objectives and quality standards.

• Review draft specifications to identify ambiguity, contradiction, missing information, and inconsistent terminology.

• Identify instructions that could lead to conflicting interpretations among raters.

• Revise guideline sets to ensure reliable application with minimal escalation.

• Document improvements to written requirements and evaluation instructions before and after revisions.

• Transform specialist-domain specifications into guidance suitable for raters across finance, retail, insurance, legal, sports, and other fields.

• Collaborate with subject matter experts to ensure accurate domain-specific terminology and professional judgment.

• Maintain technical nuance while ensuring instructions are comprehensible to non-specialist evaluators.

• Ensure a consistent structure and quality across domain-specific guideline sets.

• Participate in onboarding, calibration, documentation reviews, and ongoing refinement of guidelines.

⛳️ Requirements

• A minimum of 3 years of professional experience in linguistics, instructional design, technical writing, content design, or a closely related field.

• Direct experience in developing or refining guidelines and rubrics for human evaluators in generative AI, RLHF, or model-assessment programs.

• Proven experience in developing rater guidelines or rubrics specifically for generative AI or RLHF programs is essential.

• Demonstrated capability to resolve ambiguity and contradictions within complex written specifications.

• Experience in translating specialist requirements into clear and actionable instructions.

• Ability to work effectively across diverse subject-matter domains.

• A portfolio or specific examples that showcase measurable improvements to guidelines, rubrics, or instructional materials.

• Evidence of professional growth and increasing levels of responsibility.

• Reliable availability for a minimum of 35 hours per week during weekdays.

• Equivalent professional experience in AI evaluation, technical documentation, or guideline development may also be accepted.

• Applicants should be ready to present concrete examples of improvements made to guidelines or specifications.

• Immediate availability is preferred.

• Experience in supporting large language model evaluation, reinforcement learning from human feedback, or AI training-data programs.

• Familiarity with annotation platforms, human-feedback workflows, and rater calibration processes.

• Experience in developing domain-specific guidance for finance, insurance, retail, legal, sports, or similar fields.

• Knowledge of controlled language, information architecture, taxonomy design, or content governance.

• Experience conducting usability tests for guidelines or analyzing inter-rater consistency.

• Familiarity with version control, documentation systems, and structured authoring tools.

• Previous collaboration with researchers, program managers, engineers, and subject matter experts.

🏝️ Benefits

• Full-time remote engagement.

• Competitive hourly compensation ranging from $40–$60 per hour, depending on expertise and project scope.

• W-2 contingent employment arrangement.

• Remote work opportunity from anywhere within the United States.

• Expected commitment of at least 35 hours per week during weekdays.

• Opportunities for onboarding and calibration.

• Project scope and duration may be adjusted based on program requirements and performance.