CareerPlan

Machine Learning Scientist - ML Research

SF Bay AreaFull-time2025-12-18 → 2026-11-19
Based in United States

Core

Design and analyze experiments to evaluate AI model behavior, develop new metrics/methodologies, and analyze large-scale human voting data to uncover insights into model performance and user preferences.

Role type

Senior IC machine learning scientist (AI evaluation & research)

Builds

Scientific foundations for understanding AI at scale, including public leaderboards and tools for model developers

Domain

Artificial Intelligence / Machine Learning / Human-Centered Evaluation

Deliverable

production ML models

Required skills

Large-scale model training (reward/preference models, RLHF, DPO, contrastive learning), ML and statistics, experimental design, Python, PyTorch/JAX/TensorFlow, statistical rigor, open-source research

Preferred skills

Publishing research, working with real-world usage data, translating research to practical systems

Technologies

PyTorch, JAX, TensorFlow, Transformers, diffusion models, reinforcement learning

Responsibilities

Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions; Develop new metrics, methodologies, and evaluation protocols; Analyze large-scale human voting and interaction data; Collaborate with engineers to implement and scale research findings; Prototype and test research ideas rapidly; Author internal reports and external publications; Partner with model providers to shape evaluation questions

Seniority

Senior, hands-on IC