Product Lead, AI/ML (Evals)
Core
Lead the strategy and execution for the evals platform that enables product teams to measure AI quality, run experiments, and deploy models reliably in healthcare.
Role type
Senior Product Manager (AI/ML Evals Platform)
Builds
Shared measurement infrastructure, LLM judges, rule-based evaluators, human annotation pipelines, and online monitoring systems for model quality.
Domain
Healthcare + Generative AI / Machine Learning
Deliverable
production ML models
Required skills
Product strategy for ML platforms, evaluation framework design, annotation pipeline management, cross-functional leadership, technical fluency in distributed systems, model selection strategy
Preferred skills
ML observability systems, regulated healthcare environments, domain-specific model adaptations, personalization systems, human-in-the-loop workflows
Technologies
LLM judges, rule-based evaluators, data pipelines, distributed systems
Responsibilities
Drive product strategy and roadmap for the eval lifecycle; build shared measurement infrastructure for quality and experiments; enable fast model selection and evaluation of frontier models; define operating models and eval gates across pods; collaborate with Data Engineering, Data Science, and Clinical Science teams
Seniority
Senior, hands-on IC