Applied Scientist III, AGI Responsible AI (RAI)
Core
Design and develop rigorous evaluation methods, benchmarks, and metrics to measure the safety, fairness, robustness, and trustworthiness of large language models and generative AI.
Role type
Senior Applied Scientist (Responsible AI)
Builds
Scalable automated evaluation pipelines and evaluation systems for model launch decisions
Domain
Artificial General Intelligence (AGI), Large Language Models, Responsible AI
Deliverable
production ML models
Required skills
Machine learning model building, neural deep learning methods, statistical analysis, experimentation design, research publication, mentoring (via careerplan.io/jobs/10571444-applied-scientist-iii-agi-responsible-ai-rai-at-amazon)
Preferred skills
Large scale distributed systems, modeling tools (R, scikit-learn, Spark MLLib, MxNet, Tensorflow, numpy, scipy)
Technologies
Java, C++, Python, Spark, Hadoop, Tensorflow, numpy, scipy
Responsibilities
Lead design of evaluation frameworks for safety, fairness, and harmful content; Build automated evaluation pipelines including human-in-the-loop; Partner with pretraining and product teams to translate results into model improvements; Conduct rigorous experimentation and publish research; Mentor junior scientists; Champion responsible AI practices across the lifecycle