Staff Applied Scientist - Dashboards
Core
Define and guarantee the quality of Datadog's AI-native Dashboards product, focusing on evaluation strategies, measurement systems, and agent success metrics.
Role type
Staff Applied Scientist (AI Evaluation & Measurement)
Builds
Evaluation datasets, golden traces, regression harnesses, and measurement systems for AI agents.
Domain
Observability, GenAI, AI Evaluation
Deliverable
production ML models
Required skills
ML system evaluation at scale, experimentation design, product-driven ML leadership, cross-functional technical leadership, ambiguity navigation, metric definition (offline/online, quality/cost)
Preferred skills
GenAI initiative leadership, tool-selection accuracy optimization, retrieval relevance improvement
Technologies
Dashboards, AI agents, visualization tools
Responsibilities
Own the evaluation strategy for Dashboards and sister teams; Build eval datasets, golden traces, and regression harnesses; Drive improvements to retrieval relevance, tool-selection accuracy, and context efficiency; Provide technical leadership through design reviews and mentorship.