Principal UX Researcher
Core
Developing and scaling human-centered model evaluation approaches for AI systems to assess user needs in realistic workflows.
Role type
Principal UX Researcher (AI Evaluation)
Builds
Representative eval sets, grading rubrics, and scalable research frameworks for AI model development.
Domain
Artificial Intelligence / Machine Learning Evaluation
Deliverable
production ML models
Required skills
Mixed-methods research design, statistical analysis, experimental design, rubric development, human grading calibration, AI/ML evaluation expertise, program leadership, cross-functional partnership, tool automation, strategic planning
Preferred skills
LLM-as-a-judge methods, psychometrics, scripting/automation, telemetry integration, community of practice building
Technologies
Telemetry, experimentation platforms, data pipelines, visualization tools
Responsibilities
Designing human grading and calibration research to improve evaluation reliability; partnering with applied scientists to build eval sets and diagnose misalignment; establishing Office-wide research frameworks and quality standards; creating reusable tools and processes for scalable model evaluation; influencing technical decisions through research findings and storytelling.
Seniority
Principal, strategy & mentorship
