Senior Technical Product Manager, AI Evaluations & Quality
Core
Own technical product strategy and roadmap for AI evaluation frameworks, quality controls, safety guardrails, and model adoption to ensure safe, accurate AI outputs in professional workflows.
Role type
Senior Technical Product Manager (AI Safety & Evaluation)
Builds
Shared AI evaluation tooling, safety controls, guardrails, and benchmark datasets
Domain
AI/ML product strategy, safety, and quality assurance in professional legal/tax/government software
Deliverable
production ML models
Required skills
AI/ML product strategy, technical roadmap ownership, model evaluation, safety guardrails, data-driven decision making, technical requirements, Agile/CI/CD, cross-functional leadership
Preferred skills
Experience with LLMs, prompt engineering, security compliance, mentorship
Technologies
APIs, cloud-based distributed systems, data infrastructure, CI/CD, observability
Responsibilities
Shape AI evaluation standards and benchmark datasets, evaluate models/prompts/agents for accuracy and safety, translate quality goals into testable requirements, lead AI safety priorities with Legal/Privacy, test safeguards against prompt injection, provide evidence of quality/risks for release decisions, adopt evaluation services via documentation/onboarding, evaluate build/buy/partner options, mentor other PMs
Seniority
Senior, hands-on IC
