Staff Applied Scientist - Agentic Interfaces
Core
Define evaluation metrics and build measurement systems for AI agents interacting with Datadog's observability and security data to ensure quality, relevance, and cost-efficiency.
Role type
Staff Applied Scientist (AI Agent Evaluation & Measurement)
Builds
Evaluation datasets, golden traces, regression harnesses, and measurement platforms for AI agent integrations.
Domain
AI Agents, Observability, Software Delivery, GenAI
Deliverable
production ML models | dashboards & analysis
Required skills
ML/GenAI evaluation strategy, large-scale experimentation, product-driven research leadership, cross-functional technical leadership, ambiguity navigation, tool-selection optimization, multi-turn agent evaluation, hallucination control, cost/quality tradeoff analysis
Preferred skills
Experience with third-party agent integrations (e.g., Claude Code, Copilot), open-source agent ecosystem contributions, public speaking/technical writing
Technologies
MCP Server, Bits SRE, Bits Assistant, Bits Dev Agent, telemetry data
Responsibilities
Own the evaluation strategy for AI agent integrations; build reusable eval datasets and regression harnesses; drive improvements in retrieval relevance and tool-selection accuracy; run applied research on agent–data interaction problems; partner with internal agent teams; provide technical leadership and mentorship across the organization.