Senior Manager, Evaluation, Services Special Projects
Core
Lead a team of applied scientists and ML engineers to build evaluation methods, datasets, and automated grading systems for a conversational AI product, ensuring quality before and after release.
Role type
Senior Manager, Evaluation (Hands-on IC with team leadership)
Builds
Automated grading systems, evaluation datasets, rubrics, and quality signals for LLM-based conversational experiences
Domain
AI/ML, Conversational AI, Search & Ranking
Deliverable
production ML models
Required skills
Machine learning evaluation, LLM-based autograders and rubrics, conversational AI evaluation, statistics and experimental design, team leadership, stakeholder management
Preferred skills
Agentic or tool-use system evaluation, relevance and ranking evaluation, groundedness and factuality evaluation, logging infrastructure, synthetic data generation, privacy-preserving evaluation
Technologies
LLMs, A/B testing, automated grading systems
Responsibilities
Design and run evaluation for conversational AI layers (interpretation, retrieval, ranking, action execution), set technical direction for LLM autograders, partner with engineering to ship fixes, present quality status to senior leadership, mentor and recruit team members
Seniority
Senior, hands-on IC with management
