Principal Applied Scientist for Copilot Evals
Core
Define scientific strategy for customer-grounded quality across Copilot intents, translating user research and production incidents into evaluation and post-training priorities.
Role type
Principal Applied Scientist (AI Evaluation & Post-Training)
Builds
Reusable evaluation, regression, RLE, and post-training assets for high-value workflows and failure patterns.
Domain
Enterprise AI, Large Language Models, Copilots, Agentic Systems
Deliverable
production ML models
Required skills
Machine learning, natural language processing, information retrieval, reinforcement learning, experimentation, evaluation methods, statistical inference, experimental design, model output inspection, qualitative-to-quantitative translation, statistical sampling, measurement validity, bias analysis, uncertainty quantification, RLHF, direct preference optimization, instruction tuning, fine-tuning, human-preference data programs, enterprise grounding, offline-to-online metric correlation.
Preferred skills
Publication record, patents, demonstrated industry impact in applied research.
Technologies
Large language models, copilots, agents, tool use, retrieval-augmented generation, RLHF, DPO, instruction tuning, fine-tuning.
Responsibilities
Set scientific strategy for quality across Copilot intents; translate customer feedback and incidents into evaluation priorities; develop methods to assess evaluation-set representativeness and alignment with production outcomes; design behavior and task evaluations, rubrics, and graders; attribute quality losses across system components; mentor scientists and engineers; influence evaluation strategy across organizational boundaries.
Seniority
Principal, strategy & mentorship
