Principal Applied Scientist
Core
Lead the development of a rigorous evaluation framework for enterprise AI agents, including user simulation, experiment execution, and statistical analysis to quantify quality vs. cost trade-offs.
Role type
Principal Applied Scientist (AI Agent Evaluation & Optimization)
Builds
Production-quality evaluation tools, LLM-powered user simulators, isolated experiment execution systems, and automated insight dashboards.
Domain
Enterprise AI, Large Language Models (LLMs), Agent Evaluation, Inference Optimization
Deliverable
production ML models
Required skills
LLM and AI agent evaluation methodologies, experimental design and statistical inference, multi-objective optimization, LLM routing and inference optimization, observability and tracing, Python services and containerized environments
Preferred skills
Publications in LLM evaluation, experience with LangFuse/LangSmith, regulated enterprise environment experience
Technologies
Python, LLM gateways, containerized environments, data pipelines
Responsibilities
Design end-to-end evaluation frameworks for planner-executor agents; Develop LLM-powered user simulators; Build isolated experiment execution systems; Implement optimization plug-ins (caching, model cascading); Instrument model calls for tracing and cost attribution; Calculate KPIs (success rates, cost per task, latency); Translate results into financial impact; Build automated insight engines and API-first dashboards; Lead staged experimentation campaigns; Mentor engineers in evaluation practices.
Seniority
Principal, technical leadership & strategy