Machine Learning Engineer (Evals and Voice Models)
Core
Build evaluation foundations, benchmark design, and LLM-as-judge systems for Aircall's AI Voice Agents and messaging products to ensure accuracy, safety, and quality across voice, chat, and SMS interactions.
Role type
Senior Machine Learning Engineer (Evals and Voice Models)
Builds
Evaluation pipelines, benchmark suites, LLM-as-judge systems, and quality monitoring infrastructure for AI agents.
Domain
AI/ML, Voice Technology, Conversational AI
Deliverable
production ML models
Required skills
ML engineering, model evaluation (supervised/unsupervised/LLMs), failure analysis, voice model training (TTS/ASR), automated evaluation system design, data pipeline construction, LLM-as-judge calibration, regression testing, metric definition.
Preferred skills
LLM evaluation for agentic systems, synthetic data generation, prompt engineering, model distillation, low-latency inference optimization.
Technologies
TTS, ASR, speech-to-speech, LLMs, RAG, synthetic data tools.
Responsibilities
Design comprehensive evaluation frameworks for AI agents; train and fine-tune voice models; assess AI-generated solutions; analyze system design decisions; design annotation guidelines and calibrate LLM-as-judge systems; build live quality monitoring for deployed agents; own metric contracts; build release gates and offline regression suites; build voice-specific evaluation simulations.
Seniority
Senior, hands-on IC