AI Red Teamer – LLM Generalist

This is a fully remote position, open to applicants in United States.

📋 Description

• Conduct stress tests on large language models by deliberately attempting to break them.

• Create innovative, adversarial prompts that reveal unsafe content, biases, ineffective guardrails, hallucinations, vulnerabilities to prompt injection, and unanticipated behaviors.

• Assess models in various areas including content safety, CBRN, cybersecurity, persuasion and influence operations, child safety, self-harm, over-dependence, and regulatory compliance.

• Evaluate text, image, voice, and agentic model capabilities as project requirements dictate.

• Develop multi-turn scenarios to rigorously test AI safety measures.

• Identify ways to bypass safety filters, restrictions, and defenses using techniques such as jailbreak, evasion, and prompt injection.

• Investigate edge cases to elicit disallowed, harmful, or inaccurate outputs.

• Assess and score model responses using structured harm taxonomies and severity scales.

• Document experiments meticulously, detailing methods, reasoning, and results.

• Review and enhance adversarial prompts created by team members.

• Contribute to the advancement of harm taxonomy, calibration exercises, and inter-rater reliability initiatives.

• Collaborate with engineers, data scientists, and researchers to disseminate findings and bolster defenses.

• Regularly handle potentially distressing content.

• Keep abreast of the latest jailbreaks, attack strategies, and evolving model behaviors.

• Support Handshake AI's collaboration with premier AI research laboratories to enhance model safety and resilience.

⛳️ Requirements

• Extensive hands-on experience with various LLMs, including ChatGPT, Claude, Gemini, and open-source models.

• An innate ability to craft adversarial prompts; knowledge of jailbreak or evasion techniques is a significant advantage.

• Creative and adversarial problem-solving abilities.

• Clear and thoughtful written communication skills.

• Strong ethical judgment and the capacity to distinguish adversarial thinking from personal morals.

• Self-motivated, collaborative, and at ease in feedback-rich environments.

• A sense of curiosity, persistence, and comfort with frequent setbacks in experimentation.

• Candidates must be capable of professionally and sustainably engaging with harmful material.

• Familiarity with Python or other scripting languages.

• Experience with LLM APIs or evaluation tools.

• Competence in structured data annotation and rubric-based scoring.

• Previous work in trust and safety, content moderation, QA, or security research.

• Expertise in a high-risk field such as cybersecurity, chemistry, biology, medicine, law, or finance.

• Ability to work remotely from the United States, Monday through Friday, for 40 hours per week.

• Must be authorized to work lawfully in the United States for Handshake.

• Must not require employment visa sponsorship, as specified in the application guidelines.

🏝️ Benefits

• Support resources are available for managing exposure to disturbing content.

• Cash compensation ranges from $32 to $95 per hour.

• Opportunity for remote work from the United States.

• Full-time commitment of 40 hours per week.