CareerPlan

Machine Learning Engineer (Evals and Voice Models)

San Francisco OfficeFull-time$181k–$250k2026-11-11 → 2026-11-19
Based in United States

Core

Build evaluation foundations, benchmark design, and LLM-as-judge systems for Aircall's AI Voice Agents and messaging products to ensure accuracy, safety, and quality across voice, chat, and SMS interactions.

Role type

Senior Machine Learning Engineer (Evals and Voice Models)

Builds

Evaluation pipelines, benchmark suites, LLM-as-judge systems, and quality monitoring infrastructure for AI agents.

Domain

AI/ML, Voice Technology, Conversational AI

Deliverable

production ML models

Required skills

ML engineering, model evaluation (supervised/unsupervised/LLMs), failure analysis, voice model training (TTS/ASR), automated evaluation system design, data pipeline construction, LLM-as-judge calibration, regression testing, metric definition.

Preferred skills

LLM evaluation for agentic systems, synthetic data generation, prompt engineering, model distillation, low-latency inference optimization.

Technologies

TTS, ASR, speech-to-speech, LLMs, RAG, synthetic data tools.

Responsibilities

Design comprehensive evaluation frameworks for AI agents; train and fine-tune voice models; assess AI-generated solutions; analyze system design decisions; design annotation guidelines and calibrate LLM-as-judge systems; build live quality monitoring for deployed agents; own metric contracts; build release gates and offline regression suites; build voice-specific evaluation simulations.

Seniority

Senior, hands-on IC