Data Scientist II, AWS - Quick
Core
Engineer high-quality datasets for evaluation, benchmarking, and Responsible AI safeguards to ensure Generative AI is enterprise-ready, safe, and effective for Amazon Quick Suite.
Role type
Senior IC data scientist (evaluation & benchmarking)
Builds
Evaluation and benchmarking datasets, ground truth QA pairs, annotation guidelines, and scalable data pipelines for AI features.
Domain
Generative AI / Enterprise AI / Data Quality
Deliverable
production ML models
Required skills
Data querying (SQL), scripting (Python), machine learning/statistical modeling, AI system evaluation, statistical analysis, data pipeline development, annotation guideline creation, LLM usage (LLM-as-a-judge), synthetic data generation.
Preferred skills
Ph.D. in STEM, ML concepts for reasoning, benchmark definition for GenAI, cross-disciplinary project experience, quantitative business analysis.
Technologies
SQL, Python, R, SAS, Matlab, LLMs
Responsibilities
Design and develop comprehensive evaluation and benchmarking datasets; Leverage LLMs for synthetic data corpora generation and quality assessment; Create ground truth datasets with high-quality question-answer pairs; Lead human annotation initiatives and model evaluation audits; Develop and refine annotation guidelines and quality frameworks; Conduct statistical analysis to measure model performance and identify failure patterns; Build scalable data pipelines and tools for continuous evaluation.
Seniority
Senior, hands-on IC