About the job
Core
Evaluate and improve the quality of AI search and knowledge management services by analyzing user queries, responses, and LLM-generated answers.
Role type
AI Search Quality Analyst
Builds
High-quality evaluation datasets and insights for LLM fine-tuning and service optimization
Domain
AI / Search / Knowledge Management
Deliverable
production ML models
Required skills
User intent analysis, Answer accuracy evaluation, LLM fine-tuning data construction, Service monitoring, Data analysis, Problem definition
Preferred skills
Generative AI usage experience, LLM data labeling experience, Search/recommendation system understanding, Systematic AI answer quality analysis
Technologies
LLM, AI Search, Knowledge Management Systems
Responsibilities
Evaluate appropriateness of user queries and responses, Detect errors in AI-generated answers, Build and manage data for LLM fine-tuning, Monitor service quality for user experience optimization, Analyze issues from a user perspective for major AI services
Seniority
Junior to Mid-level, hands-on IC
