Sr Data Scientist
Core
Design evaluation datasets, metrics, and frameworks to measure the performance and safety of Amazon Quick's generative AI features and intelligent agents.
Role type
Senior Data Scientist (Evaluation & Benchmarking)
Builds
High-quality evaluation datasets, metrics, and benchmarking frameworks for enterprise AI agents
Domain
Generative AI, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Enterprise AI
Deliverable
production ML models
Required skills
Data querying (SQL), Scripting (Python), Statistical software (R, SAS, Matlab), Data science leadership, Human evaluation design, Inter-annotator agreement analysis, Model performance analysis, Gap identification, Roadmap prioritization
Preferred skills
Data visualization (AWS QuickSight, Tableau, R Shiny), Data pipeline management, LLM evaluation, RAG system evaluation, Agentic AI application evaluation
Technologies
SQL, Python, R, SAS, Matlab, AWS QuickSight, Tableau, R Shiny, LLMs, RAG systems
Responsibilities
Design and build evaluation and benchmark datasets; Define quality metrics and evaluation frameworks including human and model-based evaluation; Run human annotation programs from start to finish; Analyze model and agent performance to find gaps and root causes; Partner with science, engineering, and product teams to turn evaluation results into launch decisions; Present findings and recommendations to senior leadership
Seniority
Senior, hands-on IC with leadership responsibilities