Staff Data Scientist, Agent Improvement Loop
Core
Lead the development of evaluation strategies, analyze production data, and transform insights into measurable improvements for AI-powered agents.
Role type
Staff Data Scientist (AI Agent Evaluation & Improvement)
Builds
Evaluation systems, agent improvement cycles, and conversational AI solutions
Domain
Generative AI / Conversational AI / LLMs
Deliverable
production ML models
Required skills
LLM evaluation design, LLM-as-a-judge methodologies, Python, SQL, TypeScript, software engineering practices, GenAI systems, agent frameworks (LangGraph), LLM observability (LangSmith), causal inference, experimentation methods, MLOps/LLMOps
Preferred skills
NLP, clustering techniques, AI safety principles, mentoring experience
Technologies
Python, TypeScript, LangGraph, LangSmith, Vertex AI, Amazon SageMaker, MLflow
Responsibilities
Own and develop evaluation strategy for AI agents; Conduct continuous analysis of agent traces to identify failure patterns; Drive the complete agent improvement cycle; Improve agent skills and orchestration frameworks; Partner with Delivery/Customer Success to improve workflows; Design and validate LLM-as-a-judge systems; Develop production-quality code for evaluation tools; Mentor data scientists and engineers; Contribute to strategic planning
Seniority
Staff, hands-on IC with mentorship
