Research Scientist, Human Data & Evaluation

This is a fully remote position, open to applicants in California.

📋 Description

• Transform vague client inquiries into precise research questions, hypotheses, and study frameworks.

• Create comprehensive research proposals and methodologies from the ground up.

• Discover new research avenues and assist in determining what Contra Labs should assess, investigate, and develop.

• Formulate strategies for human data collection, evaluation protocols, rubrics, annotation schemes, and experimental designs.

• Develop strategies for prompts aimed at eliciting, assessing, and comparing model behaviors.

• Establish methods for sampling, evaluator selection, calibration, quality control, and reliability.

• Craft methodologies across various creative and multimodal fields, including design, image, video, and agentic workflows.

• Analyze both qualitative and quantitative evaluation data.

• Clarify the reasons behind observed patterns and their implications.

• Integrate findings into actionable recommendations, research narratives, and hypotheses.

• Generate client-ready reports, presentations, benchmarks, and research publications.

• Take ownership of research engagements from initial scoping to final delivery.

• Collaborate with project leads, creative professionals, research engineers, and data/engineering teams.

• Uphold research integrity and data quality within tight client deadlines.

• Manage multiple concurrent projects at various stages of completion.

• Engage in client scoping and proposal development activities.

• Team up with Strategic Project Leads for client deliverables.

⛳️ Requirements

• Located in San Francisco, California.

• 3–5+ years of pertinent research experience.

• Master's, PhD, or equivalent industry research expertise in HCI, Human-Centered AI, ML, design research, or a related discipline.

• Proficient in both qualitative and quantitative research methodologies.

• Experience in human evaluation, experimentation, data gathering, UX/HCI, annotation, or model behavior research.

• Capability to independently design and conduct rigorous research studies.

• Competence in determining what data to collect, the rationale behind it, and how to analyze it effectively.

• Skill in transforming data and observations into valuable insights, research trajectories, and actionable recommendations.

• Proficient in data analysis using Python and/or SQL.

• Exceptional written communication skills and ability to articulate methodological trade-offs and research decisions.

• Comfortable navigating ambiguity while balancing research integrity with the speed and pragmatism of a startup environment.

• Familiarity with contemporary ML and generative AI systems.

• Knowledge of RLHF, post-training, preference data, and model evaluation techniques.

• Experience in researching image, video, multimodal, creative, or agentic systems.

• Background in designing prompts or systematically investigating prompting/model behavior.

• Experience with large-scale human data or annotation initiatives.

• Proven experience working closely with AI research or model teams.

• Strong publication history or evidence of generating original research concepts.

• Experience in startup environments or 0-to-1 projects.

🏝️ Benefits

• Medical, Dental, Vision Benefits.

• 401k Matching.

• Company laptop provided on the start date.

• Equity.