CareerPlan

Expert Talent Pool, Evals and Benchmarks

RemoteFull-time2026-11-15 → 2026-11-16
Remote

Core

Design, validate, and grade benchmarks for AI research projects to assess model performance and reliability.

Role type

Senior IC AI evaluation expert (benchmarks & evals)

Builds

Rigorous benchmark test sets, gold-standard answers, and model comparison studies

Domain

Artificial Intelligence / Machine Learning Evaluation

Deliverable

production ML models

Required skills

Benchmark design, rubric creation, grader calibration, model failure analysis, user study design, Python, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)

Preferred skills

Psychometrics, QA, test engineering, UX research, data science, deep subject matter expertise

Responsibilities

Design benchmarks with scope and scoring methods; write and review rubrics; develop gold-standard answers; lead grading rounds and calibrate graders; analyze model failures and recommend fixes; conduct model comparison studies; validate benchmarks for contamination and ambiguity; design user studies on model output ratings

Seniority

Senior, hands-on IC