Product Manager, Toast IQ Evaluation
Core
Define quality standards for Toast IQ, an AI assistant for restaurant operators, by analyzing real usage sessions and building evaluation datasets.
Role type
Product Manager (AI Evaluation & Quality)
Builds
Labeled evaluation datasets, quality standards, and regression test cases for the Toast IQ AI system.
Domain
AI/LLM evaluation + Restaurant Technology
Deliverable
production ML models
Required skills
Restaurant operations experience, data analysis, failure mode analysis, dataset labeling, AI system evaluation, product prioritization, technical writing
Preferred skills
Familiarity with Toast's POS ecosystem, AI coding tools, product management experience
Technologies
Databases, AI coding tools, LLM systems
Responsibilities
Review real Toast IQ sessions to identify operator pain points; Group observations into failure modes and prioritize them; Create and maintain labeled evaluation datasets; Validate fixes in real sessions and build regression cases; Lead reviewer calibration sessions; Surface product decisions based on evaluation findings; Build tools to automate analysis and improve review workflows