Principal Research Scientist
Core
Design and own the evaluation methodology for AI agents, building realistic test environments and measurement systems to help buyers choose software with confidence.
Role type
Principal Research Scientist (AI Agent Evaluation)
Builds
Independent, rigorous evaluations of AI agents and production-ready components for G2's software insights platform.
Domain
B2B Software Industry + AI/ML Research
Deliverable
production ML models
Required skills
Evaluation design for multi-agentic systems, LLM-based agent evaluation, experimental design and statistical analysis, team mentoring, technical direction, product integration, benchmarking knowledge, clear technical writing
Preferred skills
Research in user simulation or human/AI judging, open-source evaluation tools, business software workflow familiarity, publications at top ML/NLP venues
Technologies
LLMs, LLM-based agents, automated checks, LLM-based judges, human review systems
Responsibilities
Own the evaluation methodology and set standards for testing AI agents; Design realistic evaluation environments and task sets reflecting actual business work; Develop sound measurement approaches including automated and human-based scoring; Validate evaluations by auditing tasks and quantifying noise; Mentor researchers and engineers on evaluation design and methodology; Represent evaluation findings to buyers, customers, and the research community; Set and pursue a long-term research agenda on agent evaluation problems
Seniority
Principal, hands-on IC with mentorship responsibilities
