Senior / Staff AI Engineer — Evals & Model Quality
Core
Own the evaluation frameworks and infrastructure that measure the accuracy, quality, and reliability of AI agents across workflows, document processing, and computer use.
Role type
Senior IC AI Engineer (Evals & Model Quality)
Builds
Evaluation frameworks, golden datasets, automated grading systems, and production evaluation pipelines
Domain
Insurance industry + LLMs and AI agents
Deliverable
production ML models
Required skills
Python engineering, building production AI/ML systems, LLM evaluation methodologies, model benchmarking, agent trajectory evaluation, building automated grading systems
Preferred skills
LLM-as-judge, human evaluation systems, open-weight models, fine-tuning, post-training, model routing, B2B SaaS experience
Technologies
Python, LLMs, agent architectures
Responsibilities
Build evaluation frameworks and infrastructure for AI agents; Develop golden datasets and benchmarks; Design evaluations for multi-step agent trajectories; Develop automated grading systems; Integrate evaluations into development and release processes; Continuously benchmark frontier and open-weight models
Seniority
Senior, hands-on IC
