Senior Applied Scientist, Frontier AI Assets - Assessments (FAI)
Core
Design and build automated systems for assessing the quality of datasets and benchmarks that train Amazon's frontier AI models and agents.
Role type
Senior Applied Scientist (AI Safety & Assessment)
Builds
Automated assessment systems (LLM-as-a-Judge to self-improving agents) and expert audit programs
Domain
Artificial General Intelligence (AGI), Frontier AI, Machine Learning, Data Quality
Deliverable
production ML models
Required skills
Neural deep learning, Large Language Model fundamentals, Machine learning, Data mining, Information retrieval, Statistics, Natural language processing, System architecture, Java, C++, Python
Preferred skills
Public benchmark evaluation, Large-scale human data collection, Model post-training pipelines, Recursive self-improving agents, Top-tier ML publication record, Academic research experience (via careerplan.io/jobs/10567940-senior-applied-scientist-frontier-ai-assets-assessments-fai-at-amazon)
Technologies
LLM-as-a-Judge, Self-improving agents, Neural networks
Responsibilities
Design assessment methodology for frontier AI assets, Build automated assessment systems, Oversee expert audit program, Design measurement methods for quality findings, Coach expert auditors and junior scientists, Publish research on assessment methodology
Seniority
Senior, hands-on IC with mentorship