CareerPlan

Principal Applied Scientist

US - RemoteFull-time$200k–$300k2026-11-12 → 2026-11-16

Core

Design and build an evaluation framework for an enterprise planner-based agent, replacing instinct with data-driven decisions on model selection, executor choice, and inference optimization levers to measure quality and dollar impact.

Role type

Principal Applied Scientist (AI Evaluation & Agent Architecture)

Builds

A scenario-driven user simulator, isolated experiment execution harness, statistical comparison engine, and a locally hosted dashboard for automated insights.

Domain

Artificial Intelligence, Large Language Models (LLMs), Agent Systems, Enterprise Software

Required skills

LLM and agent evaluation methods, experimental design and statistical inference, multi-objective optimization, LLM routing and inference internals, observability and tracing, Python services and containerization

Preferred skills

Publications in LLM evaluation, building internal dashboards, experience with LLM gateways (LangFuse/LangSmith), regulated enterprise domain experience

Technologies

Python, LLM gateways, LangFuse, LangSmith, containerization, data pipelines

Responsibilities

Design end-to-end evaluation workflows including scenario schemas and user simulation; build isolated experiment execution and grading systems; compute metrics like verified success, cost per success, and latency; convert differences into projected dollar savings; build an automated insight engine and locally hosted dashboard; partner with client engineers to define success criteria and explain findings.

Seniority

Principal, hands-on IC with technical leadership