Content Specialist III – AI Evaluation, Prompting

This is a fully remote position, open to applicants in United States.

📋 Description

• Test new versions of AI models across diverse topics, use cases, and conversation styles.

• Evaluate model outputs against predetermined rubrics, guidelines, and quality standards.

• Identify and document specific instances of both effective and ineffective model behavior.

• Create and enhance system prompts that influence model personality, tone, and behavior.

• Determine if evaluation rubrics and quality standards accurately assess model performance.

• Conduct quality reviews of evaluations from both human and agent-based sources to ensure precision and compliance.

• Engage in hands-on experiments with AI models and products.

• Investigate model failures and clearly document the findings.

• Consistently apply detailed instructions and evaluation criteria across a high volume of tasks.

• Adapt swiftly as product priorities, model behaviors, and evaluation requirements evolve.

• Balance quality, speed, volume, and independent judgment while adhering to detailed instructions.

• Convert observations regarding AI responses into clear and actionable feedback.

⛳️ Requirements

• Over 5 years of experience in writing, editing, journalism, production, linguistics, STEM, coding, policy, or another relevant subject area.

• At least 1 year of hands-on experience with AI preferred, including prompting, annotation, evaluation, or red teaming.

• Bachelor’s degree or equivalent professional experience.

• In-depth expertise in at least one subject area, with the capability to evaluate content as a subject matter expert.

• Strong judgment and attention to detail, with the ability to consistently apply detailed rubrics, verify information, and identify quality issues.

• Experience in assessing outputs from generative AI or large language models.

• Strong fact-checking and research abilities.

• Capability to verify claims against original sources.

• Experience in conducting thorough failure investigations.

• Excellent written communication and documentation skills.

• Ability to promptly identify and report questions and blockers.

• Comfortable with shifting priorities in response to changing product needs.

• Availability to work 40 hours per week, Monday to Friday, in an 8-hour shift.

• Proficiency in the English language is required.

• Must meet current or future US work sponsorship requirements.

🏝️ Benefits

• Dental insurance.

• Health insurance.

• Health savings account.

• Life insurance.

• Paid time off.

• Retirement plan.

• Vision insurance.

• 8-hour shift.

• Monday to Friday.