Senior/Staff Evaluation Data Analyst (Agentic Search)
Core
Own the quality measurement and improvement cycle end-to-end for AI agent search, including designing evaluation systems, benchmarks, and automated pipelines.
Role type
Senior/Staff Evaluation Data Analyst (Agentic Search)
Builds
Evaluation systems, datasets, benchmarks, metrics, and automated pipelines for AI agent search quality.
Domain
AI/ML, Agentic Search, Cloud Infrastructure
Deliverable
production ML models
Required skills
Python, experimental design, statistical analysis, root cause analysis, product roadmap prioritization, data pipeline engineering, metric definition, quality trade-off assessment
Preferred skills
Experience with complex tech products (search, AI agents, autonomous driving), ambiguity management, fast-moving environment adaptability
Technologies
Python
Responsibilities
Design and evolve evaluation systems including datasets and automated pipelines; Evaluate response quality across relevance, completeness, freshness, and factual accuracy; Investigate quality regressions and metric anomalies; Prioritize product and engineering roadmap based on quality trade-offs; Collaborate with engineers and product managers to improve outcomes.
