Director, Evaluation and Quality Assurance, Services Special Projects
Core
Build and lead a 0-to-1 evaluation and quality assurance function for a multi-year engineering initiative, defining metrics, automation, and processes to measure quality across the entire product end-to-end.
Role type
Director, Evaluation and Quality Assurance
Builds
Automated grading systems, evaluation infrastructure, and quality pipelines for conversational AI and ranking systems
Domain
AI/ML product quality, conversational AI, recommendation systems
Deliverable
production ML models
Required skills
LLM-based autograder design, human evaluation methodology, A/B testing, conversational AI evaluation, API integration QA, logging infrastructure, organizational leadership, metric definition
Preferred skills
15+ years in data science/QA, leading multi-level QA organizations
Technologies
LLMs, autograders, A/B testing frameworks, logging systems
Responsibilities
Design scalable automated grading systems and human evaluation programs; wire evaluation results into release processes as quality gates; prevent quality regressions across product surfaces; partner with engineering/ML teams to define quality standards; build evaluation organization and tooling from scratch; make resource allocation decisions under uncertainty
