AI Agent Evaluator

This is a fully remote position, open to applicants in Argentina.

📋 Description

• Assess the effectiveness of AI assistants in understanding users, utilizing personal context, and facilitating meaningful conversations.

• Craft prompts that address questions, interests, or needs pertinent to you.

• Engage in natural, multi-turn dialogues with two AI assistants.

• Incorporate questions that reference information or activities from connected services where applicable.

• Evaluate the competency of each assistant in utilizing available information and linking relevant context.

• Determine if responses are perceived as helpful, natural, trustworthy, and safe.

• Analyze how each assistant reacts to feedback and corrections.

• Offer clear justifications to support your evaluations.

• Conduct verification of currently utilized digital services and recent activities.

⛳️ Requirements

• Proficiency in one of the supported languages/locales: Spanish — Latin America; Arabic; Portuguese — Brazil; Japanese — Japan; Turkish — Turkey; Indonesian — Indonesia; Hindi — India; Korean — Korea; French — France; German — Germany.

• Excellent reading comprehension and the ability to articulate feedback clearly.

• Keen attention to detail and sound judgment in comparing conversational experiences.

• Eagerness to engage with AI assistants while utilizing relevant personal context and connected services.

• Recent experience across a qualifying assortment of digital services.

• Completion of project eligibility verification along with the necessary expert requirements.

• Familiarity with using AI assistants for everyday inquiries, planning, or research is a plus.

• Prior experience in AI evaluation, user research, or quality assurance is a plus.

• Capability to discern subtle differences in relevance, personalization, and conversation quality is a plus.

🏝️ Benefits

• Independent contractor / Part-time arrangement.

• Project length of up to 3 months.

• Opportunity to work remotely.

• Chance to contribute to the training and evaluation of cutting-edge AI models.

• Possibility of being considered for other AI evaluation opportunities if not selected for this project.