Senior/Staff Evaluation Data Analyst (Agentic Search)
Core
Own the end-to-end quality measurement and improvement cycle for AI agent search, including designing evaluation systems, building benchmarks, and diagnosing regressions to ensure agents are grounded in verified real-world data.
Role type
Senior/Staff IC Evaluation Data Analyst (AI Agents)
Builds
Evaluation systems, datasets, benchmarks, metrics, and automated pipelines for AI agent search quality.
Domain
AI infrastructure / Agentic Search / Cloud Platform
Deliverable
production ML models
Required skills
Python, experimental design, statistical analysis, root cause analysis, product roadmap prioritization, ambiguity management, engineering mindset, analytical judgment, system reasoning.
Preferred skills
Experience with complex tech products (search, AI agents, voice, autonomous driving), startup pace adaptability.
Technologies
Python
Responsibilities
Design and evolve evaluation systems including datasets and automated pipelines; Evaluate response quality across relevance, completeness, freshness, and factual accuracy; Investigate quality regressions and metric anomalies to identify root causes; Collaborate with engineers and product managers to improve outcomes; Prioritize product and engineering roadmap based on quality trade-offs.
