AI-Agent Quality & Ops Engineer (m/w/d)
Core
Ensuring measurable quality, controlled evolution, and reliable operation of agent-based AI systems for tax and accounting workflows.
Role type
AI-Agent Quality & Operations Engineer
Builds
Agent-based AI systems for tax and accounting software (ADDISON, SBS-Software, AKTE)
Domain
Tax & Accounting / AI Agents
Deliverable
production ML models
Required skills
Software engineering, quality assurance for non-deterministic systems, LLM/agent workflow evaluation strategies, regression detection across prompt/model versions, Kotlin/Java (JVM), Python, Observability (Langfuse), anomaly detection, metric-driven thinking
Preferred skills
Experience with AI coding agents, automated test generation, analysis assistants
Technologies
Kotlin, Java, Python, Langfuse
Responsibilities
Define and maintain quality criteria for agent-based workflows; build and operate evaluation suites with golden test cases and production trace replay; manage lifecycle of prompts, models, and agent tools with versioning and controlled rollouts; implement anomaly detection for quality degradation and cost anomalies; define AI-specific release readiness criteria; utilize AI-supported development workflows
Seniority
Mid-Senior, hands-on IC