Staff Technical Product Manager, Evals
Core
Own evals for an AI coding agent to make it cheaper, faster, and more accurate.
Role type
Staff Technical Product Manager (AI Evals)
Builds
Hercules Bench (offline eval set) and Hercules online evals
Domain
AI coding agents, software engineering productivity
Deliverable
production ML models
Required skills
Evals creation and management, reward modeling, system design, API design, context and harness engineering, TypeScript, React, AI SDK, eval platforms (e.g., LangSmith)
Preferred skills
None stated
Technologies
TypeScript, React, AI SDK, Postgres, Redis, Clickhouse, Tailwind, IaC (CDK, Terraform), Docker, Otel, Hono, Drizzle ORM, Temporal, Cloudflare, AWS, OpenAI, Anthropic, Google
Responsibilities
Own Hercules Bench offline eval set, own Hercules online evals, leverage 500k+ users and 1M+ code repos to guide eval creation, anonymize data to protect customer privacy
Seniority
Staff, hands-on IC