Senior AI Evals Engineer (AI Quality & Safety) - Libra-Legal AI Assistant (m/w/d)
Core
Build and own the eval platform (datasets, judges, harnesses) for a legal AI assistant to measure quality and automate optimization against targets.
Role type
Senior IC AI Evals Engineer (Quality & Safety)
Builds
Self-service eval platform, automated optimization loops, guardrails for legal AI
Domain
Legal technology + Generative AI
Deliverable
production ML models
Required skills
Python, FastAPI, LLM evaluation design, LLM-as-judge, automated optimization (GEPA/DSPy), data privacy, system architecture
Preferred skills
Data science toolkit (embedding clustering), legal domain expertise, AI coding agents
Technologies
Python, FastAPI, Langfuse, GEPA, DSPy
Responsibilities
Build self-service eval platform; Map quality landscape per task/jurisdiction; Design rubrics and LLM-as-judge criteria; Deep-dive results to automate fixes; Unhobble optimizer for automated search; Optimize cost-per-quality; Design and red-team guardrails
Seniority
Senior, hands-on IC
