Research Engineer, Intern – Benchmarking
Core
Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning; build and operate LLM evaluation systems end-to-end; run systematic failure analysis on model outputs to inform training and data pipelines.
Role type
Research Engineer (AI Evaluation & Benchmarking)
Builds
Benchmarking pipelines, evaluation systems, failure analysis workflows, rubrics, and automated evaluators
Domain
Artificial Intelligence / Large Language Models / Model Evaluation
Deliverable
production ML models
Required skills
Applied research, model evaluation, benchmarking, failure analysis, coding, ML models, data structures, algorithms, backend systems, APIs, SQL/NoSQL, cloud platforms
Preferred skills
Post-training or evaluation/benchmarking team experience, top-tier publications (NeurIPS, ICML, ACL), LLM evaluation pipelines, synthetic data generation, rubric design, RL-style workflows
Technologies
LLMs, APIs, SQL, NoSQL, Cloud platforms
Responsibilities
Design and maintain benchmarks for tool use and agentic behavior; Build and operate LLM evaluation systems; Run systematic failure analysis on model outputs; Create and refine rubrics and automated evaluators; Quantify data usability and quality; Collaborate with AI researchers and data producers
Seniority
Intern
