AI Data Analyst
Core
Evaluate AI outputs, label and categorize failure modes, and maintain high-quality benchmark data to ensure realistic, consistent datasets tied to measurable product quality lift.
Role type
Senior IC AI Data Analyst (Human Evaluation & Quality)
Builds
Human evaluation systems, benchmark datasets, and quality signals for Glean's AI product.
Domain
Enterprise AI / Work AI / LLM Evaluation
Deliverable
production ML models
Required skills
Analytical judgment, attention to detail, structured qualitative analysis, basic SQL/data retrieval, spreadsheet workflows, experience evaluating AI-generated outputs, familiarity with NLP/search/ML systems
Preferred skills
Experience with QA, data labeling, or related fields
Technologies
SQL, spreadsheets, LLM-as-a-judge tools
Responsibilities
Run ongoing human labeling for priority AI workflows; triage bad-query reports and categorize failure modes; perform qualitative analysis to identify recurring patterns; follow and improve labeling guidelines; maintain high-quality labeled datasets and monitor coverage/drift; participate in calibration exercises and validate LLM-as-a-judge outputs; partner with engineers and PMs on quality reads and release readiness; contribute to recurring quality reporting.
Seniority
Mid-Senior, hands-on IC