AI Data Analyst
Core
Human evaluation specialist ensuring AI product quality by labeling outputs, analyzing failure modes, and maintaining benchmark datasets for an enterprise Work AI platform.
Role type
Senior IC AI Data Analyst (Human Evaluation & Quality)
Builds
High-quality labeled datasets, golden sets, and regression slices for AI model evaluation and product release readiness.
Domain
Enterprise AI / Workforce AI / NLP / Agentic AI
Deliverable
production ML models
Required skills
Human evaluation of AI outputs, qualitative analysis of failure modes, data labeling, SQL/data retrieval, rubric development, calibration of LLM-as-a-judge, cross-functional collaboration
Preferred skills
Experience with NLP, search, or recommendation systems, familiarity with enterprise SaaS ecosystems
Technologies
SQL, spreadsheets, LLM-as-a-judge frameworks, enterprise SaaS platforms (Teams, Zoom, ServiceNow)
Responsibilities
Run human labeling for priority AI workflows; triage bad-query reports and categorize failure modes; perform qualitative analysis to identify patterns like hallucinations or retrieval failures; maintain high-quality labeled datasets and monitor data drift; partner with engineers and PMs on release-readiness decisions; contribute to quality reporting on failure modes and coverage gaps.
Seniority
Mid-Senior, hands-on IC
