Senior Software Engineer, AI Evaluation Infrastructure
Core
Build infrastructure for frontier AI labs to evaluate coding agents and language models, including benchmark datasets, agent evaluation runs at scale, and automated task quality checks.
Role type
Senior IC backend engineer (AI evaluation infrastructure)
Builds
Benchmark datasets, agent evaluation runs, automated task quality checks, and evaluation harnesses for frontier AI labs.
Domain
Artificial Intelligence / Machine Learning Evaluation
Deliverable
production ML models
Required skills
Python, Django, PostgreSQL, Celery, LLM evaluation, task quality judgment, system design, tradeoff analysis, ownership, TypeScript, React
Preferred skills
Sandbox/containerized environments (Docker, Kubernetes, Modal, Daytona), LLM-as-a-judge, rubric-based grading, coding agents (Claude Code, Codex, OpenHands), distributed job scheduling, open-source contributions
Technologies
Python, Django, PostgreSQL, Celery, TypeScript, React, Docker, Kubernetes, Modal, Daytona
Responsibilities
Design and ship end-to-end features (data model, API, background workers, UI); Build and improve agent/model evaluation systems; Improve benchmark task quality checks; Integrate new models/agents/benchmarks; Maintain system performance and reliability; Review pull requests
Seniority
Senior, hands-on IC
