Member of Technical Staff (Data Scientist/Engineer, Online Metrics)
Core
Derive online signals from user interactions to bridge the gap between changes in answer quality and observed user behavior for an LLM-first search engine.
Role type
Data Scientist/Engineer (Online Metrics)
Builds
Data pipelines calculating quality metrics at scale for Search, Product, and model training teams
Domain
Search, LLM-based products, Answer Quality
Deliverable
production ML models | dashboards & analysis
Required skills
Python, SQL, statistical analysis, A/B testing design, BI tools, agentic coding workflows
Preferred skills
Apache Spark, Databricks, LLM-as-a-judge systems development
Technologies
Python, SQL, Apache Spark, Databricks, BI tools
Responsibilities
Discover and validate online signals from user interactions; Design and implement novel online metrics for A/B testing and product health dashboards; Analyze experimental results to validate metrics; Build and maintain data pipelines for metric calculation; Communicate findings to Product and Search teams
Seniority
Mid-Senior, hands-on IC