CareerPlan

AI Agent Evaluation Engineer

Canada - AB (Remote)Full-time2026-11-12 → 2026-11-16

Core

Define quality metrics for Cresta's AI Receptionist, build automated evaluation systems (LLM judges, golden sets), and analyze production conversations to drive agent improvements.

Role type

Senior IC AI Agent Evaluation Engineer

Builds

Automated evaluation pipelines, golden sets, regression suites, and failure taxonomies for conversational AI agents

Domain

AI / Conversational AI / Customer Experience

Deliverable

production ML models

Required skills

LLM evaluation methodology, Python programming, conversation analysis, rubric design, statistical sampling, failure mode classification, multilingual evaluation

Preferred skills

Experience with voice/speech recognition, prompt engineering, cross-functional collaboration

Technologies

Python, LLM APIs, automated evaluation frameworks

Responsibilities

Design evaluation rubrics across adherence, outcome, and experience axes; build and calibrate automated judges against human labels; analyze production calls to cluster failures; maintain golden sets and regression suites; report quality trends with uncertainty metrics

Seniority

Senior, hands-on IC