Senior Applied Scientist - Outlook Science Team
Core
Develop and execute comprehensive strategy for LLM evaluation, encompassing quality, costs, performance, utility, and responsible AI considerations.
Role type
Senior Applied Scientist (LLM Evaluation & Strategy)
Builds
Automated evaluation pipelines integrated into product workflow; cost-effective solutions like SLMs, finetuned models, and ensemble models.
Domain
Large Language Models (LLM), Artificial Intelligence, Responsible AI
Deliverable
production ML models | dashboards & analysis | research
Required skills
LLM evaluation, program management, data science, experimentation design, metrics creation, synthetic data creation, RAG implementation, agentic workflows, Python development, responsible AI principles
Preferred skills
Small language models (SLMs), reinforcement learning, user experience optimization, deep research initiatives
Technologies
Python, LLMs, RAG, agentic workflows
Responsibilities
Develop and execute LLM evaluation strategy; oversee large-scale cross-functional evaluation programs; drive engineering product roadmap for automated pipelines; lead model and agent performance assessment; collaborate with UX teams to optimize AI interactions; implement Responsible AI guidelines; lead deep research initiatives; communicate findings to executive leadership.
Seniority
Senior, hands-on IC with strategic leadership
