Principal Research Scientist

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of the evaluation methodology and establish standards for assessing AI agents.

• Collaborate with engineers and product managers to enhance evaluations in G2’s current offerings.

• Define features, transform methods into production-ready components, and refine them based on customer and business requirements.

• Create realistic environments and task sets that mirror business operations.

• Develop precise, reproducible, and challenging scoring methods utilizing automated checks, LLM-based judges, and human reviews.

• Assess the consistency and reliability of evaluations.

• Review tasks and graders, quantify variability, and determine when agent differences are significant.

• Present and advocate for evaluation findings to buyers, customers, partners, and the research community.

• Mentor and assist researchers and engineers in evaluation design and measurement.

• Review work and document standards for reproducible evaluations.

• Establish and pursue a research agenda encompassing agent evaluation, user simulation, LLM-as-judge reliability, voice and multimodal agents, long-horizon workflows, and test set contamination.

• Stay updated on new benchmarks, papers, and evaluation practices, translating them into product and roadmap strategies.

• Draft methodology documentation, papers, and public posts when appropriate.

• Collaborate with data and go-to-market teams regarding the presentation and utilization of results.

⛳️ Requirements

• Proven experience in evaluation design or dataset curation within multi-agent systems.

• Over 5 years of experience in building or evaluating ML systems, including practical work with LLMs or LLM-based agents.

• At least 2 years of experience managing small to medium-sized teams is preferred.

• A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics, or a related field, or equivalent applied research experience.

• Demonstrated track record of original contributions to machine learning, evidenced by first-author publications or projects led from start to finish.

• Ability to independently manage a research agenda and undertake long-term projects.

• Proven history of delivering ML work in a product environment alongside engineers and product managers.

• Experience in setting technical direction, mentoring researchers or engineers, and reviewing methodologies.

• Familiarity with AI benchmarking and understanding of benchmark limitations.

• Strong statistical judgment, including expertise in experimental design, variance, significance, and agreement between human and automated raters.

• Sound judgment regarding practical trade-offs.

• Excellent writing skills and the ability to communicate methodology to non-specialists and defend it against skeptical audiences.

• Experience in agent evaluation, user simulation, or human and AI-based judging is advantageous.

• Background in conversational, voice, retrieval-based, or tool-using AI systems is beneficial.

• Contributions to open-source evaluation tools or public leaderboards are a plus.

• Familiarity with business software workflows is advantageous.

• Publications in venues such as ACL, EMNLP, NeurIPS, ICML, ICLR, COLM, or similar are beneficial.

🏝️ Benefits

• Equity

• Flexible work

• Ample parental leave

• Unlimited PTO