Senior Data Scientist for Copilot Evals
Core
Build and operate a data-driven prioritization framework for Copilot quality forums to guide quality investments, define and maintain evaluation taxonomies, and measure how well evaluation portfolios represent production traffic and customer segments.
Role type
Senior Data Scientist (AI Product Quality & Evaluation)
Builds
Quality gates, recurring scorecards, and automated pipelines connecting offline evaluation results with online customer outcomes.
Domain
AI/LLM product quality, customer feedback analysis, and data-driven product strategy.
Deliverable
production ML models | product features | dashboards & analysis
Required skills
statistical analysis, experimentation, sampling, segmentation, measurement, data visualization, SQL, Python or R, causal inference, clustering, text analytics, embeddings, classification, anomaly detection
Preferred skills
LLM or agent evaluation, DSAT analysis, model telemetry, responsible AI, privacy-preserving analysis, data governance
Technologies
SQL, Python, R, GPU platforms, CPU platforms
Responsibilities
Define and maintain intent, sub-intent, and loss-pattern taxonomies for consistent evaluation; Design quality gates with success thresholds and reporting cadences; Analyze offline-online agreement and detect meaningful quality changes; Develop resource-optimized data-analysis solutions for noisy signals; Translate customer evidence into valid task distributions and reward signals; Communicate complex analysis recommendations to senior leaders.
Seniority
Senior, hands-on IC
