Überwachung und Optimierung moderner KI-Agentensysteme im Produktivbetrieb (m/w/d)
Core
Developing best-practice approaches to monitor and optimize production AI agent systems based on transformer models using tracing tools and metrics.
Role type
Master's thesis researcher (AI system monitoring)
Builds
Tracing tools and evaluation frameworks for AI agent systems
Domain
Artificial Intelligence / Machine Learning / Software Engineering
Deliverable
research (via careerplan.io/jobs/2451601-uberwachung-und-optimierung-moderner-ki-agentensysteme-im-produktivbetrieb-mwd-at-mm-softw)
Required skills
Tracing tools analysis, metric generation, LLM-as-a-Judge, system optimization, data source integration
Preferred skills
User feedback analysis, heuristic metrics, transformer-based systems
Technologies
MLFlow, Transformers, LLM-as-a-Judge
Responsibilities
Analyze state-of-the-art tracing tools and methods, derive actionable recommendations, evaluate traces using automated metrics and user feedback, link data sources and evaluation methods
Seniority
Master's student (Research Assistant)
