Expert Talent Pool, Evals and Benchmarks
Core
Design, validate, and grade benchmarks for AI research projects to assess model performance and reliability.
Role type
Senior IC AI evaluation expert (benchmarks & evals)
Builds
Rigorous benchmark test sets, gold-standard answers, and model comparison studies
Domain
Artificial Intelligence / Machine Learning Evaluation
Deliverable
production ML models
Required skills
Benchmark design, rubric creation, grader calibration, model failure analysis, user study design, Python, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)
Preferred skills
Psychometrics, QA, test engineering, UX research, data science, deep subject matter expertise
Responsibilities
Design benchmarks with scope and scoring methods; write and review rubrics; develop gold-standard answers; lead grading rounds and calibrate graders; analyze model failures and recommend fixes; conduct model comparison studies; validate benchmarks for contamination and ambiguity; design user studies on model output ratings
Seniority
Senior, hands-on IC