Senior Machine Learning Engineer, Agent Oversight
Core
Building observability tools, evaluation frameworks, and ML systems to ensure production AI agents perform reliably and improve over time.
Role type
Senior Machine Learning Engineer (Agent Oversight)
Builds
Observability infrastructure, evaluation metrics, drift/anomaly detection systems, and improvement loops for agentic AI
Domain
Generative AI, Agentic Systems, Enterprise AI, Defense
Deliverable
production ML models
Required skills
Production ML engineering, LLM and agent architecture design, evaluation methodology design, statistical experimentation, system instrumentation, cross-functional collaboration
Preferred skills
RLHF/SFT workflows, model optimization, published research in agentic systems, regulated industry experience
Technologies
LLMs, agent orchestration frameworks, monitoring tools, statistical analysis libraries
Responsibilities
Build observability tools to track agent behavior in production; Design and automate evaluation methodologies and metrics at scale; Develop ML systems to detect drift, anomalies, or misalignment; Run rigorous experiments to validate performance improvements; Collaborate with software engineers to productionize research; Partner with product and customer teams to translate requirements into platform capabilities