Senior Backend Engineer, AI Evaluations
Core
Build infrastructure and tooling for an AI evaluations platform to enable reliable, repeatable evaluation of AI systems in production, including automated testing, debugging, and observability tools.
Role type
Senior Backend Engineer (AI Infrastructure)
Builds
Evaluation platforms, API/data integrations, test configuration management, reporting tools, and debugging/observability tools for AI components.
Domain
Artificial Intelligence / Machine Learning Infrastructure
Deliverable
production ML models
Required skills
Backend system design, API development, data pipelines, test automation, CI/CD integration, observability tooling, debugging, Python/TypeScript/Java
Preferred skills
Multi-component AI systems (ASR, LLMs, TTS), regulated environments (HIPAA), conversational AI/voice agents, LLM observability frameworks
Technologies
Python, TypeScript, Java, Langfuse, prompt eval frameworks
Responsibilities
Build and maintain infrastructure for the AI evaluations platform; Develop and productionalize evaluation frameworks for system components; Partner with ML/QA teams to translate requirements into robust systems; Improve developer experience for evaluation tools; Ensure production standards for reliability and performance.
