Machine Learning Scientist - ML Research
Core
Design and analyze experiments to evaluate AI model behavior, develop new metrics/methodologies, and analyze large-scale human voting data to uncover insights into model performance and user preferences.
Role type
Senior IC machine learning scientist (AI evaluation & research)
Builds
Scientific foundations for understanding AI at scale, including public leaderboards and tools for model developers
Domain
Artificial Intelligence / Machine Learning / Human-Centered Evaluation
Deliverable
production ML models
Required skills
Large-scale model training (reward/preference models, RLHF, DPO, contrastive learning), ML and statistics, experimental design, Python, PyTorch/JAX/TensorFlow, statistical rigor, open-source research
Preferred skills
Publishing research, working with real-world usage data, translating research to practical systems
Technologies
PyTorch, JAX, TensorFlow, Transformers, diffusion models, reinforcement learning
Responsibilities
Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions; Develop new metrics, methodologies, and evaluation protocols; Analyze large-scale human voting and interaction data; Collaborate with engineers to implement and scale research findings; Prototype and test research ideas rapidly; Author internal reports and external publications; Partner with model providers to shape evaluation questions
Seniority
Senior, hands-on IC