Senior QA Engineer, AI Agents & Agentic Workflows
Core
Own continuous validation and quality of Champion Assist, a multi-agent AI product with ~20 evolving agents, testing reasoning, tool use, orchestration, and response quality.
Role type
Senior QA Engineer (AI Agents & Agentic Workflows)
Builds
Evaluation frameworks, regression suites, and eval datasets for non-deterministic, LLM-driven systems.
Domain
Manufacturing and supply chain (SaaS)
Deliverable
production ML models
Required skills
QA experience with LLM-based applications, AI agents, or agentic workflows; designing evaluation frameworks and rubrics; building automated regression suites; testing non-deterministic outputs; understanding agentic concepts (tool/function calling, multi-agent orchestration, planning, memory/context, RAG); test automation (Selenium, Playwright, TestComplete); API testing.
Preferred skills
Experience with LLM-as-judge, golden datasets, scoring rubrics; identifying cross-agent and orchestration issues; triaging root causes in prompt/model/retrieval/tool/orchestration layers.
Responsibilities
Design and maintain end-to-end business scenarios for multi-step agentic workflows; evaluate agent outputs for correctness, relevance, groundedness, consistency, tone, and completeness; build and maintain eval datasets and automated regression suites; identify cross-agent, orchestration, and integration issues; test for hallucination, prompt injection, jailbreaks, and data leakage; detect regressions from model upgrades or prompt changes; triage and root-cause issues with Engineering and Product; re-test fixes and validate end-user experience; document test strategy and quality trends.
Seniority
Senior, hands-on IC
