AI Response Labeler – Italian Specialty

This is a fully remote position, open to applicants in Costa Rica.

📋 Description

• Conduct side-by-side evaluations of AI-generated responses to identify the stronger option.

• Assess responses for accuracy, relevance, completeness, clarity, reasoning, adherence to instructions, tone, and overall quality.

• Analyze content produced in English, Italian, or a blend of both languages.

• Evaluate a variety of tasks including general-purpose questions and answers, web-search results, file-based assignments, image-based responses, content generation requests, and both single-turn and multi-turn conversations.

• Utilize Italian expertise in language, terminology, tone, regional customs, idioms, and cultural context specific to Italy.

• Identify unsupported assertions, incomplete reasoning, overlooked instructions, awkward phrasing, cultural inaccuracies, and variances in usefulness.

• Adhere to detailed, scenario-specific annotation guidelines with accuracy and consistency.

• Make independent evaluation decisions on ambiguous cases.

• Record decisions and provide concise, evidence-based justifications when necessary.

• Accomplish evaluations within expected time and productivity benchmarks, generally completing at least 25 tasks daily.

• Engage in training, guided practice, calibration sessions, qualification assessments, and ongoing quality review activities.

• Integrate feedback and modify evaluation decisions to align with team and client quality standards.

⛳️ Requirements

• Native-level or professional fluency in Italian.

• In-depth understanding of Italian linguistic norms, regional vocabulary, idioms, tone, and cultural context specific to Italy.

• Strong fluency in English with excellent reading comprehension.

• Robust analytical and critical thinking capabilities.

• Proficient in evaluating content across diverse topics, formats, and task types.

• Capable of assessing factual accuracy, relevance, reasoning, clarity, instruction adherence, cultural appropriateness, and overall utility.

• Ability to discern subtle distinctions in meaning, quality, tone, and user intent.

• Sound judgment in applying structured evaluation criteria to ambiguous or unfamiliar situations.

• Excellent written communication skills.

• Exceptional attention to detail and the capacity to maintain accuracy within established time parameters.

• Ability to learn and consistently implement detailed evaluation frameworks.

• Capacity to work independently while adhering to shared quality standards.

• Comfortable performing repetitive, detail-oriented tasks for extended periods while maintaining focus, accuracy, and consistent judgment.

• Ability to accept feedback, adjust decisions, and adapt as evaluation guidelines evolve.

• Successful completion of an approximately 30-day onboarding and qualification program prior to commencing production work.

• Availability to work from 9:00 a.m. to 5:00 p.m. Pacific Time during the training and qualification phase.

• Preferred: experience in side-by-side labeling, annotation, comparative content evaluation, quality assessment, AI-generated response evaluation, model quality assessment, data labeling, search relevance, content quality, factual accuracy, user-facing digital experiences, detailed guidelines, rubrics, or structured decision-making frameworks.

🏝️ Benefits

• Medical, dental, and vision insurance.

• Flexible Spending Account (FSA).

• 401(k) retirement plan.

• Competitive paid time off.

• Parental leave.

• Opportunities for professional growth and development.

• Benefits in compliance with local regulations and the terms of employment through an Employer of Record partner.