AI Engineer: Evals
Core
Build the evaluation discipline and shared infrastructure for designing, deploying, and improving trustworthy AI agents in production.
Role type
Senior IC AI Engineer (Evals)
Builds
Reusable evaluation frameworks, reference implementations, golden paths, and public benchmarks for agents.
Domain
AI Agents, Engineering, Simulation, Manufacturing
Deliverable
production ML models
Required skills
Evaluation systems for agents/LLMs, dataset design, experiment design, statistical judgement, deterministic evaluators, LLM-as-a-judge, production-quality tooling, cross-team coaching, domain expert collaboration
Preferred skills
Pydantic Evals, Pydantic Logfire, engineering/simulation domain evaluation, public benchmark creation, post-training, synthetic data, regulated environments
Technologies
OpenTelemetry, Pydantic, LangGraph, LangChain, PydanticAI
Responsibilities
Lead technical strategy for agent evaluation, build reusable frameworks and golden paths, develop datasets and evaluators combining deterministic checks and human review, connect offline evaluation with production telemetry, define cross-product scorecards, create public open benchmarks, collaborate with Research and Post-Training teams, shape the team around evaluation capabilities
Seniority
Senior, hands-on IC