Expert Talent Pool, Evals and Benchmarks (Associate)
Core
Design and run assessments to measure AI model capabilities, contributing to AI safety and capability research.
Role type
Associate human-data project specialist (AI evaluation)
Builds
Benchmarks, evaluation tasks, rubrics, and reference answers for AI models
Domain
Artificial Intelligence / Machine Learning Evaluation
Deliverable
production ML models
Required skills
Question design, rubric creation, error pattern analysis, data analysis, spreadsheet proficiency, Python, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)
Preferred skills
QA/test engineering, psychometrics, UX research, ML evaluation, research support
Responsibilities
Write questions and tasks to test model capabilities; Develop rubrics defining answer quality; Create reference answers and edge cases; Grade model outputs; Analyze failed answers and document error patterns; Compare multiple models using identical prompts; Validate benchmarks; Participate in user studies
Seniority
Associate, 0-5 years experience
