AI Benchmark Engineer
Core
Design, build, and validate rigorous evaluation benchmarks (Terminal-Bench) to test large language models on multilingual software challenges, focusing on non-English data processing and locale/encoding edge cases.
Role type
Independent contractor AI benchmark engineer (multilingual software evaluation)
Builds
Verifiable evaluation suites and task environments for testing AI model robustness across languages
Domain
Artificial Intelligence / Large Language Models / Multilingual Software Engineering
Deliverable
production ML models
Required skills
Python, shell scripting, data processing, terminal/CLI development, coding agents, multilingual text processing, Unicode normalization, locale-dependent conventions, bidirectional/RTL handling
Preferred skills
Native or near-native fluency in a specific language, deep understanding of grammar/register/phrasing, experience at leading tech companies, top-tier engineering university background
Technologies
Python, shell scripting, Terminal/CLI, coding agents
Responsibilities
Create high-signal tasks in native language, find failure points in AI performance, write deterministic verifier scripts, analyze execution logs, calibrate task difficulty, participate in human quality control processes
Seniority
Mid-level, hands-on IC