CareerPlan

Member of Technical Staff (Language Model Evaluations)

San Francisco (On-site)Full-time2026-07-29 → 2026-11-18
On-site

Core

Designing and building next-generation frontier evaluations, datasets, and scoring systems for language models to measure reasoning, knowledge, coding, and agentic capabilities.

Role type

Senior IC machine-learning engineer (LLM evaluation)

Builds

Frontier benchmarks, evaluation harnesses, datasets, and the Artificial Analysis Intelligence Index

Domain

AI benchmarking and language model evaluation

Deliverable

production ML models

Required skills

Python, LLM evaluation landscape expertise, statistical analysis, dataset construction, contamination resistance strategies, agentic evaluation methods

Preferred skills

Experience with pre-release model benchmarking, published research in NLP/ML evaluation, AI-native workflow adoption

Technologies

Python, evaluation harnesses, LLMs

Responsibilities

Design and ship frontier evaluations across reasoning, knowledge, coding, and agentic capabilities; construct contamination-resistant datasets and scoring systems; benchmark leading labs' systems including pre-release models; publish influential analysis and data visualizations; run evaluation suites across major model releases; maintain result integrity

Seniority

Senior, hands-on IC