Member of Technical Staff - Evaluations
Core
Design and maintain internal evaluation suites and infrastructure to ensure post-training decisions rely on accurate, reproducible benchmarks for real-world tasks, specifically targeting small edge models and agentic workflows.
Role type
Senior IC evaluation engineer (ML infrastructure)
Builds
Internal eval suite, infrastructure, and open-source evaluation benchmarks
Domain
Artificial Intelligence, Machine Learning, Edge Computing
Deliverable
production ML models
Required skills
Eval design, benchmark construction, contamination handling, reproducibility, software engineering, infrastructure debugging, reading generated outputs, variance analysis
Preferred skills
Model training experience, publications/research background, deep systems expertise, small edge model familiarity
Technologies
lm-eval-harness, lighteval, OLMES, HELM, Inspect
Responsibilities
Work on the internal eval suite and infrastructure to keep it reliable, optimized, and reproducible at scale; Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity; Build evals targeted at small edge models; Read generated outputs to debug unexpected scores; Turn customer and open-source feedback into usable environments for data, training, and evaluation; Release open-source evals and write papers about them
Seniority
Senior, hands-on IC