Principal Applied Scientist
Core
Design and build an evaluation framework for an enterprise planner-based agent, replacing instinct with data-driven decisions on model selection, executor choice, and inference optimization levers to measure quality and dollar impact.
Role type
Principal Applied Scientist (AI Evaluation & Agent Architecture)
Builds
A scenario-driven user simulator, isolated experiment execution harness, statistical comparison engine, and a locally hosted dashboard for automated insights.
Domain
Artificial Intelligence, Large Language Models (LLMs), Agent Systems, Enterprise Software
Required skills
LLM and agent evaluation methods, experimental design and statistical inference, multi-objective optimization, LLM routing and inference internals, observability and tracing, Python services and containerization
Preferred skills
Publications in LLM evaluation, building internal dashboards, experience with LLM gateways (LangFuse/LangSmith), regulated enterprise domain experience
Technologies
Python, LLM gateways, LangFuse, LangSmith, containerization, data pipelines
Responsibilities
Design end-to-end evaluation workflows including scenario schemas and user simulation; build isolated experiment execution and grading systems; compute metrics like verified success, cost per success, and latency; convert differences into projected dollar savings; build an automated insight engine and locally hosted dashboard; partner with client engineers to define success criteria and explain findings.
Seniority
Principal, hands-on IC with technical leadership