Principal Applied Scientist - Copilot Quality
Core
Build trusted quality measures for AI assistants and large language models, including metrics, evaluation datasets, and benchmarks grounded in real customer tasks.
Role type
Principal Applied Scientist (AI Quality & Evaluation)
Builds
Large-scale AI products (LLMs, AI assistants, agents)
Domain
Artificial Intelligence / Machine Learning
Deliverable
production ML models
Required skills
Large model evaluation, deep learning, post-training, reward modeling, reinforcement learning, retrieval, ranking, experimental design, statistical analysis, A/B testing, metric development
Preferred skills
Human and AI-assisted labeling approaches, hill-climbing optimization, regression protection, rollout criteria design
Technologies
Large language models, AI assistants, agents
Responsibilities
Develop online and offline metrics and representative evaluation datasets; Validate statistical soundness of quality measures; Improve quality through post-training, retrieval, ranking, prompting, and orchestration; Design experiments to distinguish offline results from customer experience improvements; Establish rigorous A/B experimentation and quality gates.
Seniority
Principal, hands-on IC with strategy & mentorship
