AI Rater Guidelines Writer, Linguist, Instructional Designer

This is a fully remote position, open to applicants in United States.

📋 Description

• Transform intricate and unclear program requirements into straightforward, concise, and user-friendly rater guidelines.

• Create detailed evaluation rubrics and scoring frameworks that facilitate uniform decision-making across various AI tasks.

• Analyze current documentation to pinpoint ambiguities, inconsistencies, contradictions, and coverage deficiencies, then update guidelines to enhance clarity and usability.

• Translate subject-matter requirements from industries such as finance, retail, insurance, legal, and sports into structured evaluation instructions that non-domain raters can easily implement.

• Work alongside research teams, program managers, and subject matter experts to ensure consistency across multiple guideline sets.

• Produce documentation that covers edge cases, exceptions, and complex evaluation scenarios while reducing the need for reviewer escalations.

⛳️ Requirements

• At least 3 years of professional experience in Linguistics, Instructional Design, Technical Writing, AI Content Development, or a related field.

• Hands-on experience in creating, refining, or maintaining evaluation guidelines, rating rubrics, or reviewer instructions in Generative AI, RLHF, human evaluation, or AI data annotation contexts.

• Proven capability to navigate various subject areas and translate domain-specific knowledge into clear, structured documentation.

• Established experience in resolving ambiguities and enhancing written specifications with measurable improvements.

• Strong analytical skills coupled with an exceptional attention to detail.

• Evidence of career advancement and professional development.

• Ability to reliably commit to 35+ hours per week during weekdays.

• Excellent written communication skills, capable of articulating nuanced concepts in a clear, consistent, and easily understandable manner.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Opportunities for professional development and career advancement.

• Flexible working hours and the possibility of remote work.

• Comprehensive health and wellness benefits.