Voice AI Evals - Lead
Core
Define and build the measurement, feedback, and improvement system for production voice AI agents in healthcare to drive automation rates and patient satisfaction.
Role type
Senior Lead, Voice AI Evaluation & Measurement
Builds
Closed-loop evaluation systems, conversation quality rubrics, and operational reporting for FDEs
Domain
Healthcare AI, Voice Agents, Conversational AI
Deliverable
production ML models
Required skills
LLM-as-judge evaluation rubric design, conversation quality metrics definition, turn efficiency analysis, audio layer evaluation (ASR/TTS), production data systems experience, stakeholder leadership
Preferred skills
Experience with Retell or similar voice platforms, inter-rater reliability calibration, Spanish-English parity evaluation
Technologies
LLMs, Metabase, Firebase, MongoDB, Postgres, BigQuery, Langfuse, Prefect
Responsibilities
Own automation rate and patient satisfaction as operating metrics; Define and deploy conversation quality rubrics; Build turn efficiency and voice/speech evaluation; Close the evaluation loop from production calls to product fixes; Make evaluation actionable for FDEs; Lead the data team through craft
Seniority
Senior, hands-on IC with leadership responsibilities
