Member of Technical Staff (Language Model Evaluations)
Core
Designing and building next-generation frontier evaluations, datasets, and scoring systems for language models to measure reasoning, knowledge, coding, and agentic capabilities.
Role type
Senior IC machine-learning engineer (LLM evaluation)
Builds
Frontier benchmarks, evaluation harnesses, datasets, and the Artificial Analysis Intelligence Index
Domain
AI benchmarking and language model evaluation
Deliverable
production ML models
Required skills
Python, LLM evaluation landscape expertise, statistical analysis, dataset construction, contamination resistance strategies, agentic evaluation methods
Preferred skills
Experience with pre-release model benchmarking, published research in NLP/ML evaluation, AI-native workflow adoption
Technologies
Python, evaluation harnesses, LLMs
Responsibilities
Design and ship frontier evaluations across reasoning, knowledge, coding, and agentic capabilities; construct contamination-resistant datasets and scoring systems; benchmark leading labs' systems including pre-release models; publish influential analysis and data visualizations; run evaluation suites across major model releases; maintain result integrity
Seniority
Senior, hands-on IC