AI Data Quality Analyst
Core
Evaluate LLM-generated conversations and text outputs, perform data labeling, and monitor production data to ensure quality and identify trends for AI model improvement.
Role type
AI Data Quality Analyst (LLM evaluation & labeling)
Builds
Reliable datasets for model training, benchmarking, and evaluation for Workday/Paradox AI products
Domain
Generative AI / Large Language Models (LLM)
Deliverable
production ML models
Required skills
Qualitative research, pattern detection in textual data, data annotation, LLM evaluation, cross-functional communication, English fluency
Preferred skills
Data labeling experience, LLM analysis experience, AI tool usage (Claude Code, Cursor)
Technologies
Web-based tools, labeling platforms, dashboards
Responsibilities
Review and evaluate LLM-generated conversations against quality criteria, perform high-quality annotation for model training, monitor real production data for quality trends, identify qualitative patterns and failure modes, analyze evaluation results and communicate insights to stakeholders, partner with AI Engineers and Product Managers on data needs
Seniority
Individual Contributor