Staff Product Manager (Evals)
Core
Own evaluations for AI agents, establishing internal frameworks for agent quality and building customer-facing tools for builders to assess and improve their agents.
Role type
Staff Product Manager (AI Agent Evaluations)
Builds
Internal evaluation framework for Workato's AI features and customer-facing evaluation experience for builders
Domain
Enterprise iPaaS / AI Agents / LLMs
Deliverable
product features
Required skills
Product management (7+ years), Hands-on AI/ML evaluation writing, Technical product shipping, Framework adoption across engineering, B2B product sensibility, Greenfield product definition, Internal influence, Technical translation
Preferred skills
Agent architectures/RAG/LLM dev experience, ML engineering background, Developer tools/platform products, Evaluation frameworks (human eval, automated benchmarks, red-teaming)
Technologies
Agent Studio, LLMs, RAG systems
Responsibilities
Define and own the evaluation framework for internal AI agent features, Build the customer-facing evaluation experience, Make hard calls on evaluation complexity exposure, Partner with Build Experience PM, Work with ML engineers and platform teams, Establish metrics for agent quality and adoption, Spend time with customers understanding agent assessment struggles
Seniority
Staff, hands-on IC with strategic scope
