CareerPlan

Principal Applied Scientist

USRemoteFull-time$200k–$300k2026-11-13 → 2026-11-16
Remote · from United States

Core

Lead the development of a rigorous evaluation framework for enterprise AI agents, including user simulation, experiment execution, and statistical analysis to quantify quality vs. cost trade-offs.

Role type

Principal Applied Scientist (AI Agent Evaluation & Optimization)

Builds

Production-quality evaluation tools, LLM-powered user simulators, isolated experiment execution systems, and automated insight dashboards.

Domain

Enterprise AI, Large Language Models (LLMs), Agent Evaluation, Inference Optimization

Deliverable

production ML models

Required skills

LLM and AI agent evaluation methodologies, experimental design and statistical inference, multi-objective optimization, LLM routing and inference optimization, observability and tracing, Python services and containerized environments

Preferred skills

Publications in LLM evaluation, experience with LangFuse/LangSmith, regulated enterprise environment experience

Technologies

Python, LLM gateways, containerized environments, data pipelines

Responsibilities

Design end-to-end evaluation frameworks for planner-executor agents; Develop LLM-powered user simulators; Build isolated experiment execution systems; Implement optimization plug-ins (caching, model cascading); Instrument model calls for tracing and cost attribution; Calculate KPIs (success rates, cost per task, latency); Translate results into financial impact; Build automated insight engines and API-first dashboards; Lead staged experimentation campaigns; Mentor engineers in evaluation practices.

Seniority

Principal, technical leadership & strategy