CareerPlan

AI Benchmark Engineer | Native Language Specialist - Japanese - Remote

Japan (Remote)RemoteFull-time2026-02-23 → 2026-11-20
RemoteNo visa sponsorship

Core

Design, build, and validate rigorous evaluation benchmarks (Terminal-Bench) to test large language models on multilingual software challenges, specifically focusing on non-English data processing and locale/encoding edge cases.

Role type

Native language specialist / AI benchmark engineer (freelance)

Builds

High-signal multilingual benchmark tasks and deterministic verifier scripts for coding agents

Domain

AI evaluation / Multilingual software engineering / Terminal workflows

Deliverable

production ML models

Required skills

Python, shell scripting, data processing, terminal/CLI development, coding agents, multilingual text processing (encoding/decoding, Unicode, locale conventions)

Preferred skills

Native or near-native fluency in Japanese, deep understanding of grammar/register/phrasing, high English proficiency, background at leading tech companies or top-tier engineering universities

Technologies

Python, Shell, CLI, Terminal-Bench, LLMs (Haiku, Sonnet, Opus)

Responsibilities

Create realistic task environments using native language assets, find failure points in AI models within native language contexts, write reliable verifier scripts, analyze execution logs to calibrate task difficulty, participate in 4-layer human quality control process

Seniority

Mid-level, hands-on IC