Research Engineer, AI Evaluations
Core
Design, build, and run end-to-end AI evaluations (benchmarks, rubrics, leaderboards) to measure how well models serve underrepresented communities and languages.
Role type
Senior Research Engineer (AI Evaluations)
Builds
Evaluation pipelines, leaderboards, and statistical analysis tools for AI models
Domain
AI safety, evaluation, and multilingual NLP
Deliverable
production ML models
Required skills
ML engineering, Python, statistical analysis, API deployment, GPU inference, benchmark design, bias detection, error attribution
Preferred skills
Human judgment data, speech/multilingual NLP, Indian languages, publications
Technologies
Python, APIs, GPU inference, self-hosted systems
Responsibilities
Design benchmarks and rubrics; Deploy and serve models via APIs; Analyze evaluation data with statistical methods; Build modular evaluation pipelines; Prototype new evaluation methods
Seniority
Senior, hands-on IC