CareerPlan

AI Benchmark Engineer | Native Language Specialist - Portuguese (Brazil) - Remote

Brazil (Remote)RemoteFull-time2026-09-08 → 2026-11-20
RemoteFreelance / independent contractor

Core

Design, build, and validate rigorous evaluation benchmarks (Terminal-Bench) to test large language models on multilingual software challenges and locale/encoding edge cases.

Role type

Senior IC AI Benchmark Engineer (Native Language Specialist)

Builds

Verifiable evaluation suites and task environments for multilingual coding agents

Domain

AI evaluation, multilingual software engineering, terminal workflows

Deliverable

production ML models

Required skills

Python, shell scripting, data processing, terminal/CLI development, coding agents, multilingual text processing, Unicode normalization, locale-dependent conventions

Preferred skills

Experience at leading technology companies, top-tier engineering university graduation, native or near-native fluency in Portuguese (Brazil), high English proficiency

Technologies

Python, shell scripting, Terminal-Bench, Haiku, Sonnet, Opus

Responsibilities

Create high-signal tasks in native language, find failure points in AI models, write deterministic verifier scripts, analyze execution logs, calibrate task difficulty, participate in 4-layer human quality control process

Seniority

Senior, hands-on IC