Staff Machine Learning Engineer, Agent Eval Platform
Core
Build the self-improving loop for AI agents that act on enterprise systems, focusing on automated evaluation, failure analysis, and harness optimization without training the model.
Role type
Staff Machine Learning Engineer (Agent Eval Platform)
Builds
Automated evaluation infrastructure, self-improving agent harness components, and measurement architecture for multi-tenant enterprise environments.
Domain
Enterprise AI Agents, LLM Evaluation, Systems Engineering
Deliverable
production ML models
Required skills
Applied ML fundamentals, Python, production engineering, LLM-as-judge design, human annotation program management, experimentation rigor, failure analysis at scale, synthetic data generation, prompt and harness optimization, high-volume trace data pipelines
Preferred skills
Evaluated LLM agents with tool calls/actions, generated complete agent environments, published evaluation methodology, built eval tooling for external teams, worked on agentic systems
Technologies
Python, LLMs, data pipelines, simulation environments
Responsibilities
Design and calibrate shared base judges with rubrics and confidence scoring; split problems between deterministic validators and LLM judges; build infrastructure for finding failures and proposing harness improvements; verify proposed changes on unseen cases; ensure learnings are scoped per organization; own the end-to-end measurement architecture and mentor engineers on evaluation methodology.
Seniority
Staff, technical ownership & mentorship
