AI Response Labeler – Italian Specialty

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Conduct side-by-side evaluations of AI-generated outputs to identify the stronger response.

• Assess the responses for factual accuracy, relevance, completeness, clarity, reasoning, adherence to instructions, tone, and overall quality.

• Review content produced in English, Italian, or a combination of both languages.

• Evaluate general-purpose inquiries and answers, web-search outcomes, file-based tasks, image-based replies, content generation requests, as well as single-turn and multi-turn dialogues.

• Leverage Italian expertise to analyze language, terminology, tone, regional conventions, idioms, and cultural nuances pertinent to Italy.

• Detect unsupported assertions, incomplete reasoning, overlooked instructions, unnatural phrasing, cultural inaccuracies, and differences in usefulness.

• Apply precise, scenario-specific annotation guidelines consistently and accurately.

• Make independent evaluation judgments and provide brief, evidence-supported rationales.

• Fulfill evaluations within set time and productivity expectations, typically completing a minimum of 25 tasks each day.

• Engage in training, guided practice, calibration meetings, qualification reviews, and ongoing quality assessment activities.

• Integrate feedback and modify evaluation decisions to meet team and client quality standards.

• Complete onboarding and qualification before commencing production tasks.

⛳️ Requirements

• Native-level or professional fluency in Italian.

• Extensive knowledge of Italian linguistic conventions, regional vocabulary, idioms, tone, and cultural context specific to Italy.

• Strong fluency in English with excellent reading comprehension.

• Capability to comprehend complex prompts, AI-generated responses, and detailed annotation guidelines provided in English.

• Robust analytical and critical-thinking skills.

• Competence in evaluating content across diverse topics, formats, and task types.

• Ability to assess factuality, relevance, reasoning, clarity, adherence to instructions, cultural appropriateness, and overall usefulness.

• Skill in identifying subtle distinctions in meaning, quality, tone, and user intent.

• Sound judgment in ambiguous or unfamiliar situations.

• Strong written communication abilities to clearly and concisely explain evaluation decisions.

• Exceptional attention to detail and the capacity to maintain accuracy while adhering to established time expectations.

• Ability to learn and consistently apply intricate evaluation frameworks.

• Capacity to work independently while aligning with shared quality standards.

• Comfort in performing repetitive, detail-oriented tasks for extended periods while sustaining focus, accuracy, and consistent judgment.

• Ability to accept feedback, recalibrate decisions, and adapt as evaluation guidelines evolve.

• Preferred: experience with side-by-side labeling, annotation, comparative content evaluation, quality assessment, AI-generated response evaluation, model quality assessment, data labeling, search relevance, content quality, factual accuracy, user-facing digital experiences, or detailed guidelines/rubrics.

🏝️ Benefits

• Medical, dental, and vision coverage.

• Flexible Spending Account (FSA).

• 401(k) retirement plan.

• Competitive paid time off.

• Parental leave.

• Opportunities for professional growth and development.

• Benefits in accordance with local requirements and employment terms through an Employer of Record partner.