AI Evaluation Systems Engineering Lead
Core
Build evaluation infrastructure to help researchers improve frontier models and make reliable release decisions.
Role type
Senior IC engineering lead (distributed systems & ML evaluation)
Builds
End-to-end evaluation systems for benchmark configuration, model inference, grading, aggregation, and results publication
Domain
AI/ML research infrastructure, distributed systems, LLM evaluation
Deliverable
production ML models
Required skills
Distributed systems engineering, Python, C/C++/Java/JavaScript/C#, GPU-backed inference, distributed training, large-scale compute workloads, system architecture, team leadership, operational excellence, reproducibility validation
Preferred skills
LLM evaluation, benchmarking, model-based grading, reinforcement learning infrastructure, Ray/Kubernetes/Slurm, experimental reproducibility, hiring/mentoring engineers
Technologies
Ray, Kubernetes, Slurm, Python, C, C++, C#, Java, JavaScript
Responsibilities
Set technical direction and roadmap for evaluation systems; Build infrastructure for benchmarking and model evaluation; Accelerate research iteration and GPU efficiency; Ensure reproducible and trustworthy results; Maintain operational excellence and reliability; Contribute code and debug complex distributed problems; Build and mentor engineering teams; Align with research and engineering partners
Seniority
Senior, hands-on IC with leadership responsibilities