Evals Infrastructure Tech Lead / Manager
Core
Lead the team building distributed systems to schedule, orchestrate, and execute evaluations for frontier AI model training, ensuring measurement quality and throughput.
Role type
Senior IC Tech Lead / Manager (Evals Infrastructure)
Builds
Large-scale distributed eval orchestration systems, compute allocation harnesses, and reproducible evaluation pipelines for frontier models.
Domain
AI Safety / LLM Infrastructure / Distributed Systems
Deliverable
production ML models | infrastructure
Required skills
Python, Rust, large-scale distributed systems leadership, high-throughput fault-tolerant system design, cloud/on-prem accelerator fleet management, team management, research-infrastructure translation, observability, time-series regression detection.
Preferred skills
LLM inference/training infrastructure experience, agentic evals with sandboxed execution, statistical literacy (variance, confidence intervals), uncertainty quantification.
Technologies
Python, Rust, cloud/on-prem accelerator fleets, time-series metrics tools.
Responsibilities
Lead the team building distributed systems that schedule, orchestrate, and execute evals for frontier model training; Own eval throughput and cost via compute allocation and queueing; Build and scale harnesses for researchers to define and run evals; Ensure eval results are deterministic and reproducible; Ensure eval signal reaches dashboards for launch decisions; Contribute directly as an engineer while managing and growing the team.
Seniority
Senior, hands-on IC with management responsibilities