Expert Talent Pool, Evals and Benchmarks (Associate)
Core
Design and run human-led evaluations of AI models by creating rigorous questions, rubrics, and comparative assessments to measure model capabilities.
Role type
Associate AI evaluation specialist
Builds
Benchmarks, evaluation tasks, and comparative assessments for AI models
Domain
Artificial Intelligence / Machine Learning Evaluation
Deliverable
production ML models
Required skills
QA/test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, rigorous question writing, rubric development, error pattern analysis, spreadsheet/Python usage, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)
Preferred skills
Experience with model comparisons, user and rating studies
Responsibilities
Write questions and tasks that test specific model capabilities, Develop rubrics that define strong, partial, and weak answers, Create reference answers and edge-case scenarios to challenge models, Grade model outputs against rubrics and explain each score clearly, Analyze incorrect answers, group error patterns, and document findings, Compare multiple models using the same prompts, Validate benchmarks and flag questions that may no longer provide reliable measurements, Support small studies of how people evaluate or interact with model outputs
Seniority
Associate, 0-5 years experience
