Product Designer, Evals & Prompts
Core
Building automated evaluation pipelines, graders, and low-code tools to test and refine prompts for Claude's product surfaces, ensuring alignment between model behavior and user expectations.
Role type
Senior IC Product Designer (Evals & Prompts)
Builds
Automated evaluation harnesses, low-code testing tools for designers, and prompt migration strategies for model releases.
Domain
Generative AI / LLM Product Design
Deliverable
production ML models | product features
Required skills
Python, building evaluation pipelines for LLMs, creating internal tools for non-coders, standing up test harnesses, shipping prompts, analyzing transcripts
Preferred skills
Model-launch cycle experience, A/B testing, front-end development, converting product rubrics into training signals
Technologies
Python, LLM evaluation frameworks, internal tooling platforms
Responsibilities
Write and revise prompts for Claude's tools and features; build graders to automate prompt testing; develop low-code eval tools for designers; support model releases with prompt fixes and migrations; scale the eval harness across 50-100 tools; package prompt fixes for training data
Seniority
Senior, hands-on IC
