CareerPlan

Expert Talent Pool, Evals and Benchmarks (Associate)

RemoteFull-time2026-11-15 → 2026-11-16

Core

Design and run assessments to measure AI model capabilities, contributing to AI safety and capability research.

Role type

Associate human-data project specialist (AI evaluation)

Builds

Benchmarks, evaluation tasks, rubrics, and reference answers for AI models

Domain

Artificial Intelligence / Machine Learning Evaluation

Deliverable

production ML models

Required skills

Question design, rubric creation, error pattern analysis, data analysis, spreadsheet proficiency, Python, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)

Preferred skills

QA/test engineering, psychometrics, UX research, ML evaluation, research support

Responsibilities

Write questions and tasks to test model capabilities; Develop rubrics defining answer quality; Create reference answers and edge cases; Grade model outputs; Analyze failed answers and document error patterns; Compare multiple models using identical prompts; Validate benchmarks; Participate in user studies

Seniority

Associate, 0-5 years experience