AI Agent Evaluator

This is a fully remote position, open to applicants in United States.

📋 Description

• Assess recorded and simulated AI agent interactions focused on acute symptom triage, evaluating decisions for both escalation and de-escalation.

• Analyze agent dialogues concerning insulin and diabetes medication titration to ensure clinical accuracy and adherence to guidelines.

• Review agent-provided travel health advice, which includes pre-travel counseling, vaccination suggestions, and risk assessments specific to destinations.

• Detect clinical safety issues, edge cases, and deficiencies in triage logic or clinical reasoning.

• Score and annotate agent performance utilizing structured evaluation rubrics.

• Deliver actionable written feedback to clinical and product teams to enhance agent development.

• Engage in case review and calibration meetings with other clinical subject matter experts (SMEs).

• Assist in creating and refining test scenarios for ongoing agent quality assurance.

⛳️ Requirements

• Registered Nurses: must possess an active, unrestricted RN License.

• Nurse Practitioners: must hold an active, unrestricted Nurse Practitioner license in the United States, along with current board certification (ANCC or AANP).

• A minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a similar acute-care environment.

• Clinical experience in managing patients with diabetes, including insulin titration.

• Strong, up-to-date knowledge of relevant triage protocols and clinical practice guidelines.

• Capability to critically assess clinical reasoning and detect inaccuracies or safety risks in both written and recorded clinical interactions.

• Ability to work independently in a remote, asynchronous contract position.

• Excellent written communication skills for providing structured clinical feedback.

• Experience in telephone triage or nurse advice line environments (recommended).

• Experience in travel medicine, occupational health, or pre-travel consultations (recommended).

• Knowledge of CDC and other travel medicine guidelines (recommended).

• Prior experience in clinical quality assurance, utilization review, or case review (recommended).

• Familiarity with AI-assisted clinical tools or digital health platforms (recommended).

🏝️ Benefits

• Fully remote work.

• Flexible scheduling.

• Hourly compensation based on experience.