User Researcher, AI Evaluations
Core
Define and scale evaluation frameworks for Notion's AI-powered experiences, focusing on end-to-end product quality, user trust, and recovery behaviors.
Role type
Senior UX Researcher (AI Evaluations)
Builds
Reusable rubrics, longitudinal study protocols, and human-in-the-loop evaluation loops for AI agents and generative UI.
Domain
AI / Generative AI / Collaborative Workspaces
Deliverable
production ML models | product features
Required skills
UX research craft (quant + qual), AI fluency and systems thinking, operationalizing insight into measurement, clear communication and impact orientation, pragmatism in fast-moving environments
Preferred skills
LLM-as-judge methods, prompt design for evaluators, AI observability tooling, data querying languages (SQL), scripting languages (Python), statistical/mathematical software (R, SAS, Matlab)
Technologies
Dovetail, Listen Labs, Maze, Outset, Braintrust
Responsibilities
Define evaluation criteria reflecting user expectations (helpfulness, trust, tone, control, transparency); run recurring longitudinal and feature-specific surveys/studies; anchor evaluations in real workflows (jobs-to-be-done); identify failure modes and recovery behaviors; operationalize evaluation with Product, Design, Engineering, and Data Science partners
Seniority
Senior, hands-on IC