AI Agent Evaluation Engineer
Core
Define quality metrics for Cresta's AI Receptionist, build automated evaluation systems (LLM judges, golden sets), and analyze production conversations to drive agent improvements.
Role type
Senior IC AI Agent Evaluation Engineer
Builds
Automated evaluation pipelines, golden sets, regression suites, and failure taxonomies for conversational AI agents
Domain
AI / Conversational AI / Customer Experience
Deliverable
production ML models
Required skills
LLM evaluation methodology, Python programming, conversation analysis, rubric design, statistical sampling, failure mode classification, multilingual evaluation
Preferred skills
Experience with voice/speech recognition, prompt engineering, cross-functional collaboration
Technologies
Python, LLM APIs, automated evaluation frameworks
Responsibilities
Design evaluation rubrics across adherence, outcome, and experience axes; build and calibrate automated judges against human labels; analyze production calls to cluster failures; maintain golden sets and regression suites; report quality trends with uncertainty metrics
Seniority
Senior, hands-on IC