CareerPlan

AI Engineer: Evals

LondonFull-time2026-11-13 → 2026-11-16

Core

Build the evaluation discipline and shared infrastructure for designing, deploying, and improving trustworthy AI agents in production.

Role type

Senior IC AI Engineer (Evals)

Builds

Reusable evaluation frameworks, reference implementations, golden paths, and public benchmarks for agents.

Domain

AI Agents, Engineering, Simulation, Manufacturing

Deliverable

production ML models

Required skills

Evaluation systems for agents/LLMs, dataset design, experiment design, statistical judgement, deterministic evaluators, LLM-as-a-judge, production-quality tooling, cross-team coaching, domain expert collaboration

Preferred skills

Pydantic Evals, Pydantic Logfire, engineering/simulation domain evaluation, public benchmark creation, post-training, synthetic data, regulated environments

Technologies

OpenTelemetry, Pydantic, LangGraph, LangChain, PydanticAI

Responsibilities

Lead technical strategy for agent evaluation, build reusable frameworks and golden paths, develop datasets and evaluators combining deterministic checks and human review, connect offline evaluation with production telemetry, define cross-product scorecards, create public open benchmarks, collaborate with Research and Post-Training teams, shape the team around evaluation capabilities

Seniority

Senior, hands-on IC