Applied Scientist, Agent Evaluation & Adaptive Model Routing
Core
Build evaluation and decision systems to make agentic inference measurable, reliable, and adaptive, including LLM evaluation pipelines and adaptive model-routing strategies.
Role type
Applied Scientist (LLM Evaluation & Adaptive Inference)
Builds
Evaluation pipelines, task suites, adaptive routing policies, and research prototypes for agentic inference systems.
Domain
Artificial Intelligence, Large Language Models (LLMs), Agentic Systems, Cloud Infrastructure
Deliverable
production ML models
Required skills
LLM evaluation, agentic systems design, adaptive inference, Python, PyTorch, experimental pipeline construction, trajectory-level analysis, cost-quality trade-off analysis, statistical analysis, model selection and routing, uncertainty estimation, cascading and escalation, shadow testing, A/B testing, multi-model API integration, vLLM, SGLang
Preferred skills
Preference modeling, contextual bandits, online learning, multi-turn agent evaluation, tool-using agent evaluation, traffic replay, production model monitoring, open-source contributions in evaluation/routing/inference
Technologies
Python, PyTorch, vLLM, SGLang
Responsibilities
Own and extend LLM and agent evaluation pipelines; develop representative task suites; measure behavior at the trajectory level; research and prototype adaptive model-routing strategies; take promising methods from offline evaluation through traffic replay, shadow testing, and internal pilots.
Seniority
Senior, hands-on IC