Program Manager, Human Evaluations (Recommendations)
Core
Design and execute human evaluation frameworks to measure and improve the quality of Netflix's recommendation systems across the homepage and discovery experience.
Role type
Senior Program Manager, Human Evaluations
Builds
Scalable human evaluation programs, rubrics, and workflows for recommendation quality
Domain
Streaming entertainment (Film/TV) + Machine Learning
Deliverable
production ML models
Required skills
Program management, vendor management, evaluation design, rubric development, rater calibration, operational efficiency, cross-functional leadership, film/TV content fluency
Preferred skills
Personalization/recommender systems experience, offline evaluation vs A/B testing knowledge, list/page-level evaluation experience
Technologies
Human labeling platforms
Responsibilities
Lead end-to-end execution of human evaluation initiatives, partner with Product Management on strategy translation, manage human annotation vendors, drive operational efficiency, develop scalable guidelines and scoring rubrics, oversee rater onboarding and QA, pilot and refine evaluation tasks, resolve bottlenecks
Seniority
Senior, hands-on IC
