Senior Machine Learning Engineer, AI Safety Evaluations
Core
Design and maintain safety benchmarks, metrics, and automated red-teaming tools for large language and multimodal models to evaluate AI safety and risk.
Role type
Senior Machine Learning Engineer (AI Safety Evaluations)
Builds
Safety benchmarks, evaluation pipelines, automated red-teaming tools, and reproducible experiment frameworks.
Domain
Generative AI, AI Safety, Large Language Models (LLMs), Multimodal Models
Deliverable
production ML models
Required skills
Python, statistical analysis, benchmark design, experiment tracking, failure analysis, data pipeline engineering
Preferred skills
LLM evaluation, adversarial testing, human-in-the-loop workflows, statistical rigor (confidence intervals), research publications
Technologies
Python, version control, experiment tracking systems
Responsibilities
Design safety benchmarks and scoring rubrics; define evaluation metrics and protocols; build reusable Python pipelines for reproducible results; develop automated red-teaming tools; investigate model failures and document findings; communicate evaluation results to researchers and executives.
