Staff Product Manager - AI Evals
Core
Build measurable, reliable, and continuously improvable agentic AI systems by defining evaluation capabilities, performance metrics, and product quality standards.
Role type
Staff Product Manager (AI Evals)
Builds
AI evaluation tooling and workflows for agentic systems
Domain
Cybersecurity + Agentic AI
Deliverable
production ML models
Required skills
Product management for AI/agentic systems, building/managing agent evaluations, translating technical needs to product requirements, defining performance metrics, operationalizing evaluation workflows
Preferred skills
Cybersecurity/offensive cyber experience, familiarity with benchmarks/human/automated evaluation/regression testing, experience with autonomous systems + human oversight
Technologies
Agentic AI frameworks, evaluation benchmarks, trace analysis tools
Responsibilities
Own product roadmap for agentic AI evaluations, partner with engineering/AI teams on evaluation needs, define agent performance and quality measures, turn evaluation needs into product requirements, operationalize evaluation workflows, use results to inform product decisions
