CareerPlan

Member of Technical Staff - Evaluations

San FranciscoFull-time2026-11-10 → 2026-11-16
Hybrid · United States

Core

Design and maintain internal evaluation suites and infrastructure to ensure post-training decisions rely on accurate, reproducible benchmarks for real-world tasks, specifically targeting small edge models and agentic workflows.

Role type

Senior IC evaluation engineer (ML infrastructure)

Builds

Internal eval suite, infrastructure, and open-source evaluation benchmarks

Domain

Artificial Intelligence, Machine Learning, Edge Computing

Deliverable

production ML models

Required skills

Eval design, benchmark construction, contamination handling, reproducibility, software engineering, infrastructure debugging, reading generated outputs, variance analysis

Preferred skills

Model training experience, publications/research background, deep systems expertise, small edge model familiarity

Technologies

lm-eval-harness, lighteval, OLMES, HELM, Inspect

Responsibilities

Work on the internal eval suite and infrastructure to keep it reliable, optimized, and reproducible at scale; Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity; Build evals targeted at small edge models; Read generated outputs to debug unexpected scores; Turn customer and open-source feedback into usable environments for data, training, and evaluation; Release open-source evals and write papers about them

Seniority

Senior, hands-on IC