CareerPlan

Staff Machine Learning Engineer, Agent Eval Platform

Mountain View, CALIFORNIA, usFull-time$176k–$308k2026-11-13 → 2026-11-16

Core

Build the self-improving loop for AI agents that act on enterprise systems, focusing on automated evaluation, failure analysis, and harness optimization without training the model.

Role type

Staff Machine Learning Engineer (Agent Eval Platform)

Builds

Automated evaluation infrastructure, self-improving agent harness components, and measurement architecture for multi-tenant enterprise environments.

Domain

Enterprise AI Agents, LLM Evaluation, Systems Engineering

Deliverable

production ML models

Required skills

Applied ML fundamentals, Python, production engineering, LLM-as-judge design, human annotation program management, experimentation rigor, failure analysis at scale, synthetic data generation, prompt and harness optimization, high-volume trace data pipelines

Preferred skills

Evaluated LLM agents with tool calls/actions, generated complete agent environments, published evaluation methodology, built eval tooling for external teams, worked on agentic systems

Technologies

Python, LLMs, data pipelines, simulation environments

Responsibilities

Design and calibrate shared base judges with rubrics and confidence scoring; split problems between deterministic validators and LLM judges; build infrastructure for finding failures and proposing harness improvements; verify proposed changes on unseen cases; ensure learnings are scoped per organization; own the end-to-end measurement architecture and mentor engineers on evaluation methodology.

Seniority

Staff, technical ownership & mentorship