AI Product Manager, AI Platform - Quality and Evaluation
Core
Define and execute product strategy for AI quality, observability, and evaluation platforms to ensure LLM accuracy and reliability.
Role type
Senior Product Manager, AI Platform (Quality & Evaluation)
Builds
Scalable LLM evaluation frameworks, automated evaluation harnesses, AI observability tools, and quality benchmarking systems.
Domain
Artificial Intelligence / Large Language Models / Enterprise Software
Deliverable
production ML models | product features
Required skills
Product strategy for technical platforms, LLM evaluation frameworks, AI quality measurement methodologies, model benchmarking, regression testing, AI observability platforms, prompt engineering, experimentation frameworks, LLM-as-a-judge techniques, cross-functional partnership with engineers and researchers, analytical skills for translating AI signals to product decisions.
Preferred skills
Retrieval-augmented generation (RAG), AI agents, model orchestration, synthetic data generation, internal developer platforms, ML infrastructure, enterprise SaaS, financial technology.
Technologies
Datadog LLM Observability, Langfuse, LangSmith, Arize AI, Braintrust, Galileo.
Responsibilities
Define product roadmap for AI Quality Platform, build automated evaluation harnesses, establish offline/online evaluation methodologies and golden datasets, drive product decisions using AI quality metrics, partner with engineering to deliver platform capabilities, translate foundation model advances into platform features.
Seniority
Senior, hands-on IC
