CareerPlan

Expert Talent Pool, Evals and Benchmarks

RemoteFull-time2026-11-16 → 2026-11-18
Remote

Core

Design, validate, and grade benchmarks to assess AI model performance rigorously.

Role type

Senior IC AI evaluation expert (benchmarks & evals)

Builds

Benchmarks, test sets, rubrics, and grading operations for AI research projects

Domain

AI research / Machine Learning evaluation

Deliverable

production ML models

Required skills

Benchmark design, rubric creation, gold-standard answer development, grader calibration, model failure analysis, user study design, Python, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)

Preferred skills

Psychometrics, QA, test engineering, UX research, data science, subject matter expertise

Responsibilities

Design benchmarks from ground up, write and review rubrics, develop gold-standard answers, lead grading rounds, analyze model failures, design user studies

Seniority

Senior, hands-on IC