Senior
Core
Own evaluation frameworks and infrastructure measuring the accuracy, quality, and reliability of AI agents across workflows, assistants, document processing, and computer use.
Role type
Senior IC AI evaluation engineer (LLM agents)
Builds
Evaluation frameworks, golden datasets, benchmarks, regression suites, and production pipelines for insurance AI workflows
Domain
Insurance + Applied AI / LLM Agents
Deliverable
production ML models
Required skills
Python engineering, LLM architecture understanding, evaluation methodologies, model benchmarking, building production-grade systems, owning ambiguous technical problems
Preferred skills
LLM-as-judge systems, human evaluation, agent trajectory evaluations, large-scale benchmarking, open-weight models, fine-tuning, post-training, model routing, B2B SaaS 0-to-1 experience
Technologies
Python, LLMs, AI agents
Responsibilities
Develop golden datasets, benchmarks, and regression suites with Forward Deployed Engineers and domain experts for real-world insurance tasks and edge cases. Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion. Build LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and quality rubrics. Integrate offline and online evaluations into development and release processes to measure improvements and catch regressions. Benchmark frontier and open-weight models across accuracy, latency, cost, and reliability to inform model selection and production releases. Create continuous improvement loops using production failures, customer feedback, and agent traces.