Senior Machine Learning Engineer, Agent Eval Platform
Core
Build the evaluation loop and infrastructure for AI agents that act on enterprise systems, focusing on judging agent performance, calibrating LLM judges, and creating self-improving agent harnesses.
Role type
Senior Machine Learning Engineer (Agent Evaluation & Infrastructure)
Builds
Automated evaluation systems, LLM judges, synthetic data environments, and self-improving agent harnesses for enterprise AI agents.
Domain
Enterprise AI, Agentic Systems, LLM Evaluation, MLOps
Deliverable
production ML models
Required skills
Applied ML fundamentals, Python, LLM-as-judge design, automated evaluation design, human annotation program management, search ranking/recsys evaluation, fine-tuning small models, reward modeling, agent trajectory analysis, prompt engineering as an engineering discipline
Preferred skills
Experience with offline/online divergence, side-by-side rater agreement, RLHF/RLAIF, versioned prompt testing
Technologies
Python, LLMs, Synthetic data generation, Enterprise systems
Responsibilities
Design and calibrate LLM judges against human rubrics; build infrastructure for reproducible evaluation across hundreds of organizations; generate synthetic data and environments from real failures; propose and verify improvements to agent instructions, tools, and routing; maintain a loop where fixed failures become permanent tests; ensure learnings are scoped to specific organizations.
Seniority
Senior, hands-on IC