CareerPlan

Senior / Staff AI Engineer — Evals & Model Quality

San FranciscoFull-time2026-10-14 → 2026-11-20
Hybrid

Core

Own the evaluation frameworks and infrastructure that measure the accuracy, quality, and reliability of AI agents across workflows, document processing, and computer use.

Role type

Senior IC AI Engineer (Evals & Model Quality)

Builds

Evaluation frameworks, golden datasets, automated grading systems, and production evaluation pipelines

Domain

Insurance industry + LLMs and AI agents

Deliverable

production ML models

Required skills

Python engineering, building production AI/ML systems, LLM evaluation methodologies, model benchmarking, agent trajectory evaluation, building automated grading systems

Preferred skills

LLM-as-judge, human evaluation systems, open-weight models, fine-tuning, post-training, model routing, B2B SaaS experience

Technologies

Python, LLMs, agent architectures

Responsibilities

Build evaluation frameworks and infrastructure for AI agents; Develop golden datasets and benchmarks; Design evaluations for multi-step agent trajectories; Develop automated grading systems; Integrate evaluations into development and release processes; Continuously benchmark frontier and open-weight models

Seniority

Senior, hands-on IC