AI Response Labeler, Italian Specialty

This is a fully remote position, open to applicants in Chile.

📋 Description

• Conduct side-by-side evaluations of AI-generated outputs to determine the superior response.

• Assess responses for accuracy, relevance, completeness, clarity, reasoning, adherence to instructions, tone, and overall quality.

• Review content written in English, Italian, or a combination thereof.

• Evaluate general-purpose questions and answers, web-search results, file-based tasks, image-based responses, content-generation requests, and both single-turn and multi-turn conversations.

• Utilize Italian expertise in language, terminology, tone, regional conventions, idioms, and cultural nuances specific to Italy.

• Identify unsupported assertions, incomplete reasoning, overlooked instructions, awkward phrasing, cultural inaccuracies, and variations in usefulness.

• Accurately and consistently apply scenario-specific annotation guidelines.

• Make independent evaluation decisions and provide clear, evidence-based rationales.

• Complete at least 25 evaluation tasks daily while balancing productivity with quality.

• Engage in training, guided practice, calibration sessions, qualification reviews, and ongoing quality review activities.

• Incorporate feedback and adjust decisions to meet team and client quality standards.

⛳️ Requirements

• Native-level or professional proficiency in Italian.

• Comprehensive understanding of Italian linguistic conventions, regional vocabulary, idioms, tone, and cultural context specific to Italy.

• Strong fluency in English with excellent reading comprehension.

• Robust analytical and critical thinking abilities.

• Capability to evaluate content across diverse topics, formats, and task types.

• Ability to assess factual correctness, relevance, reasoning, clarity, adherence to instructions, cultural appropriateness, and overall usefulness.

• Skill in recognizing subtle differences in meaning, quality, tone, and user intention.

• Sound judgment in ambiguous or unfamiliar situations.

• Strong written communication skills.

• Exceptional attention to detail and the ability to maintain accuracy within set timeframes.

• Ability to learn and consistently apply detailed evaluation frameworks.

• Capacity to work independently while remaining in sync with shared quality standards.

• Comfort in performing repetitive, detail-oriented tasks for extended periods.

• Ability to receive feedback, recalibrate judgments, and adapt as evaluation guidelines evolve.

• Must successfully complete a structured onboarding and qualification program prior to production work.

• Preferred: experience with side-by-side labeling, annotation, comparative content evaluation, quality assessment, evaluation of AI-generated responses, model quality assessment, data labeling, search relevance, content quality, factual accuracy, user-facing digital experiences, detailed guidelines, rubrics, or structured decision-making frameworks.

🏝️ Benefits

• Medical, dental, and vision insurance coverage.

• Flexible Spending Account (FSA).

• 401(k) retirement savings plan.

• Competitive paid time off.

• Parental leave.

• Opportunities for professional growth and development.

• Benefits in accordance with local regulations and the terms of employment through an Employer of Record partner.