CareerPlan

Senior Machine Learning Engineer, Agent Eval Platform

Santa Clara, CALIFORNIA, usFull-time$161k–$274k2026-11-13 → 2026-11-16

Core

Build the evaluation loop and infrastructure for AI agents that act on enterprise systems, focusing on judging agent performance, calibrating LLM judges, and creating self-improving agent harnesses.

Role type

Senior Machine Learning Engineer (Agent Evaluation & Infrastructure)

Builds

Automated evaluation systems, LLM judges, synthetic data environments, and self-improving agent harnesses for enterprise AI agents.

Domain

Enterprise AI, Agentic Systems, LLM Evaluation, MLOps

Deliverable

production ML models

Required skills

Applied ML fundamentals, Python, LLM-as-judge design, automated evaluation design, human annotation program management, search ranking/recsys evaluation, fine-tuning small models, reward modeling, agent trajectory analysis, prompt engineering as an engineering discipline

Preferred skills

Experience with offline/online divergence, side-by-side rater agreement, RLHF/RLAIF, versioned prompt testing

Technologies

Python, LLMs, Synthetic data generation, Enterprise systems

Responsibilities

Design and calibrate LLM judges against human rubrics; build infrastructure for reproducible evaluation across hundreds of organizations; generate synthetic data and environments from real failures; propose and verify improvements to agent instructions, tools, and routing; maintain a loop where fixed failures become permanent tests; ensure learnings are scoped to specific organizations.

Seniority

Senior, hands-on IC