AI Benchmark Engineer | Native Language Specialist - Portuguese (Brazil) - Remote
Core
Design, build, and validate rigorous evaluation benchmarks (Terminal-Bench) to test large language models on multilingual software challenges and locale/encoding edge cases.
Role type
Senior IC AI Benchmark Engineer (Native Language Specialist)
Builds
Verifiable evaluation suites and task environments for multilingual coding agents
Domain
AI evaluation, multilingual software engineering, terminal workflows
Deliverable
production ML models
Required skills
Python, shell scripting, data processing, terminal/CLI development, coding agents, multilingual text processing, Unicode normalization, locale-dependent conventions
Preferred skills
Experience at leading technology companies, top-tier engineering university graduation, native or near-native fluency in Portuguese (Brazil), high English proficiency
Technologies
Python, shell scripting, Terminal-Bench, Haiku, Sonnet, Opus
Responsibilities
Create high-signal tasks in native language, find failure points in AI models, write deterministic verifier scripts, analyze execution logs, calibrate task difficulty, participate in 4-layer human quality control process
Seniority
Senior, hands-on IC
