This is a fully remote position, open to applicants in Singapore.

📋 Description

• Evaluate and test advanced language models to address challenges.

• Engage in conversations with models using real-world scenarios and evaluation prompts.

• Assess factual correctness and logical consistency.

• Develop and implement test plans and regression suites.

• Create rubrics and establish pass/fail criteria.

• Document reproducible error traces along with root-cause hypotheses.

• Propose enhancements for prompt engineering, guardrails, and evaluation metrics.

• Collaborate on adversarial red-teaming and automation using Python/SQL.

• Design dashboards to monitor quality changes over time.

• Record failure modes to enhance model reasoning and reliability.

⛳️ Requirements

• Proficiency in model evaluation, LLM safety, prompt robustness, data quality assurance, multilingual and domain-specific testing, grounding verification, and compliance/readiness evaluations.

• Familiarity with hallucination detection, factual consistency, prompt-injection and jailbreak resistance, bias/fairness audits, chain-of-reasoning reliability, tool-use accuracy, retrieval-augmentation fidelity, and end-to-end workflow validation.

• A bachelor's, master's, or PhD in computer science, data science, computational linguistics, statistics, or a related discipline is preferred.

• Experience in delivering QA for ML/AI systems.

• Background in safety/red-team operations.

• Knowledge of test automation frameworks, such as PyTest.

• Practical experience with LLM evaluation tools like OpenAI Evals, RAG evaluators, and W&B.

• Skills in designing evaluation rubrics.

• Experience in adversarial testing/red-teaming.

• Conducting regression testing at scale.

• Auditing for bias/fairness.

• Verifying grounding.

• Engineering prompts and system prompts.

• Automating tests using Python/SQL.