AI/ML Engineer
Core
Detect subtle failures and issue patterns across large volumes of production AI-agent traces using user-behavior signals and develop evaluation methods for detection quality.
Role type
AI/ML Engineer (Production AI Agents)
Builds
Patch-generation workflows, failure detection systems, and evaluation methods for AI agents.
Domain
Artificial Intelligence / Machine Learning / Production Systems
Deliverable
production ML models
Required skills
Production LLMs, LLM-as-judge approaches, embeddings, failure pattern detection, evaluation method development, patch-generation workflows, cost-effectiveness optimization, production AI agent operation
Preferred skills
Operating AI agents in production
Technologies
LLMs, embeddings
Responsibilities
Detect subtle failures and issue patterns across large volumes of production AI-agent traces, Develop evaluation methods for measuring detection quality without labeled ground truth, Build patch-generation workflows that reproduce failures and verify fixes, Select appropriate models and methods to improve the cost-effectiveness of failure detection, Regularly review production customer traces to identify improvement opportunities, Use rephrasing, abandonment, retries, and other user-behavior signals to identify unreported issues
Seniority
0-10 years experience (Entry to Senior)
