Senior Product Manager, Conversational AI Chatbot & Agent Quality
Core
Building and operating production conversational AI chatbots and agent quality systems for a crypto exchange, focusing on knowledge base architecture, retrieval quality, and agent evaluation.
Role type
Senior Product Manager, Conversational AI & Agent Quality
Builds
Production chatbot agents, knowledge bases, agent evaluation harnesses, and internal agent runtime platforms.
Domain
Cryptocurrency / Conversational AI / LLM Operations
Deliverable
production ML models | product features
Required skills
Knowledge base architecture, retrieval quality tuning, labeling pipeline design, agent evaluation harness design, automated scoring rubrics, dialogue flow design, log-based failure triage, human-in-the-loop workflow design, metrics ownership (resolution rate, CSAT), SOP-to-agent-flow translation
Preferred skills
Agent runtime product design, tool permission modeling, developer-facing observability dashboards, regression testing for non-deterministic systems, feedback-driven improvement loops
Technologies
LLMs, Retrieval Augmented Generation (RAG), Agent Runtimes, Evaluation Harnesses, Labeling Platforms
Responsibilities
Own knowledge base structure, content quality, and retrieval coverage; Design SOPs and translate them into agent dialogue flows; Build and maintain labeling pipelines and annotation guidelines; Design agent evaluation harnesses and automated scoring rubrics; Manage human-in-the-loop workflows for low-confidence cases; Own key metrics like resolution rate and CSAT.