Researcher, AI Evaluations
Core
Design and execute human evaluations, rubrics, and benchmarks to assess AI model quality, failure patterns, and uncertainty, guiding model improvements and product decisions.
Role type
Senior individual contributor AI evaluation researcher
Builds
Validated evaluation methods, rubrics, datasets, benchmarks, and automated grader validation pipelines
Domain
AI/ML research and evaluation within design collaboration tools
Deliverable
production ML models
Required skills
Designing human evaluations and scoring rubrics, building representative evaluation datasets and benchmarks, validating automated graders (including LLM-as-judge), qualitative and quantitative research analysis, statistical expertise in sampling and significance testing, stakeholder management, problem-solving in ambiguous environments
Preferred skills
Integrating automated evaluations and regression testing into engineering workflows, developing scalable evaluation frameworks, background in product design or data science, familiarity with tools like Braintrust, LangSmith, or DeepEval
Technologies
LLM-as-judge, Braintrust, LangSmith, DeepEval
Responsibilities
Define quality criteria and design studies for priority AI workstreams, build rubrics and benchmarks reflecting user needs, work with engineers to integrate validated methods into repeatable evaluations, explain quality and failure patterns to guide model selection and release decisions, validate automated graders against human ratings, connect user research with model development work
Seniority
Senior, hands-on IC