CareerPlan

Expert Talent Pool, Evals and Benchmarks (Associate)

RemoteFull-time2026-11-16 → 2026-11-18

Core

Design and run human-led evaluations of AI models by creating rigorous questions, rubrics, and comparative assessments to measure model capabilities.

Role type

Associate AI evaluation specialist

Builds

Benchmarks, evaluation tasks, and comparative assessments for AI models

Domain

Artificial Intelligence / Machine Learning Evaluation

Deliverable

production ML models

Required skills

QA/test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, rigorous question writing, rubric development, error pattern analysis, spreadsheet/Python usage, evaluation tools (Inspect, lm-eval-harness, HELM, promptfoo)

Preferred skills

Experience with model comparisons, user and rating studies

Responsibilities

Write questions and tasks that test specific model capabilities, Develop rubrics that define strong, partial, and weak answers, Create reference answers and edge-case scenarios to challenge models, Grade model outputs against rubrics and explain each score clearly, Analyze incorrect answers, group error patterns, and document findings, Compare multiple models using the same prompts, Validate benchmarks and flag questions that may no longer provide reliable measurements, Support small studies of how people evaluate or interact with model outputs

Seniority

Associate, 0-5 years experience