Principal Data Scientist - Agent Builder
Core
Define evaluation strategy and quality metrics for Elastic's conversational and agentic platform, focusing on RAG, agents, tools, and retrieval systems to improve groundedness and reliability.
Role type
Principal Data Scientist (Applied Leadership)
Builds
Elastic Search AI Platform (conversational and agentic search)
Domain
Search AI, Large Language Models (LLMs), Retrieval Augmented Generation (RAG)
Deliverable
production ML models | product features
Required skills
evaluation strategy design, offline/online experimentation, LLM-as-judge calibration, retrieval systems (dense/sparse/vector), re-ranking, query understanding, statistical rigor, Python, PyTorch/Transformers, Elasticsearch
Preferred skills
ES|QL familiarity, multimodality exploration, cost/latency trade-off analysis
Technologies
Elasticsearch, Python, PyTorch, Transformers, Pandas
Responsibilities
Define evaluation strategy including golden datasets and rubrics; Lead design of quality metrics for RAG and agents; Build and compare retrieval/re-ranking approaches; Turn experimental results into product decisions; Partner with engineering to productionize evaluation pipelines; Mentor data scientists and engineers on experiment design.
Seniority
Principal, strategy & mentorship
