Expert Talent Pool, Evals and Benchmarks
Remote
Core
Design, validate, and grade benchmarks to assess AI model performance rigorously.
Role type
Senior IC AI evaluation expert (benchmarks & evals)
Builds
Benchmarks, test sets, rubrics, and grading operations for AI research projects
Domain
AI research / Machine Learning evaluation
Deliverable
production ML models
Required skills
Benchmark design, rubric creation, gold-standard answer development, grader calibration, model failure analysis, user study design, Python, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)
Preferred skills
Psychometrics, QA, test engineering, UX research, data science, subject matter expertise
Responsibilities
Design benchmarks from ground up, write and review rubrics, develop gold-standard answers, lead grading rounds, analyze model failures, design user studies
Seniority
Senior, hands-on IC
