Content Specialist III | AI Evaluation & Prompting
Core
Evaluate, refine, and improve advanced AI models and products by testing responses, writing prompts, and assessing quality against rubrics.
Role type
Senior IC AI evaluation and prompting specialist
Builds
Improved AI product experiences and communication behaviors
Domain
Artificial Intelligence / Large Language Models
Deliverable
production ML models
Required skills
AI Model Evaluation, Prompt Writing and Refinement, Writing and Editing, Rubric Based Quality Assessment, Fact Checking, Analytical Judgment
Preferred skills
Generative AI evaluation, Failure investigation, Research skills, Source verification
Technologies
Large Language Models (LLMs), Generative AI
Responsibilities
Test new AI model versions across topics and use cases, Evaluate model responses against established rubrics, Write and refine system prompts, Assess evaluation rubrics effectiveness, Perform quality reviews of human and agent evaluations, Investigate model failures and document findings
Seniority
Senior, hands-on IC