Staff Applied Scientist - Agentic Interfaces
Core
Define evaluation metrics and build measurement systems for AI agents interacting with Datadog's observability and security data to ensure quality, relevance, and cost-efficiency.
Role type
Staff Applied Scientist (AI Evaluation & Measurement)
Builds
Evaluation datasets, golden traces, regression harnesses, and measurement substrates for first-party and third-party AI agents.
Domain
Observability, Security, Generative AI, Agent Evaluation
Deliverable
production ML models | dashboards & analysis
Required skills
ML system evaluation at scale, experimental design, metric definition, cross-functional technical leadership, product mindset, ambiguity navigation
Preferred skills
GenAI initiative leadership, tool selection optimization, multi-turn agent evaluation, open research in agent-data interaction
Technologies
MCP Server, Bits SRE, Bits Assistant, Bits Dev Agent, telemetry data
Responsibilities
Own evaluation strategy for AI agent integrations; build reusable eval datasets and regression harnesses; drive improvements in retrieval relevance and tool-selection accuracy; run applied research on agent evaluation challenges; provide technical leadership and mentorship across the organization.