CareerPlan

Staff Technical Product Manager, Evals

San FranciscoFull-time$200k–$350k2026-10-21 → 2026-11-20
Hybrid · United States

Core

Own evals for an AI coding agent to make it cheaper, faster, and more accurate.

Role type

Staff Technical Product Manager (AI Evals)

Builds

Hercules Bench (offline eval set) and Hercules online evals

Domain

AI coding agents, software engineering productivity

Deliverable

production ML models

Required skills

Evals creation and management, reward modeling, system design, API design, context and harness engineering, TypeScript, React, AI SDK, eval platforms (e.g., LangSmith)

Preferred skills

None stated

Technologies

TypeScript, React, AI SDK, Postgres, Redis, Clickhouse, Tailwind, IaC (CDK, Terraform), Docker, Otel, Hono, Drizzle ORM, Temporal, Cloudflare, AWS, OpenAI, Anthropic, Google

Responsibilities

Own Hercules Bench offline eval set, own Hercules online evals, leverage 500k+ users and 1M+ code repos to guide eval creation, anonymize data to protect customer privacy

Seniority

Staff, hands-on IC