CareerPlan

Senior QA Engineer, AI Agents & Agentic Workflows

Pune, MH, inRemoteFull-time2026-11-05 → 2026-11-19
Hybrid

Core

Own continuous validation and quality of Champion Assist, a multi-agent AI product with ~20 evolving agents, testing reasoning, tool use, orchestration, and response quality.

Role type

Senior QA Engineer (AI Agents & Agentic Workflows)

Builds

Evaluation frameworks, regression suites, and eval datasets for non-deterministic, LLM-driven systems.

Domain

Manufacturing and supply chain (SaaS)

Deliverable

production ML models

Required skills

QA experience with LLM-based applications, AI agents, or agentic workflows; designing evaluation frameworks and rubrics; building automated regression suites; testing non-deterministic outputs; understanding agentic concepts (tool/function calling, multi-agent orchestration, planning, memory/context, RAG); test automation (Selenium, Playwright, TestComplete); API testing.

Preferred skills

Experience with LLM-as-judge, golden datasets, scoring rubrics; identifying cross-agent and orchestration issues; triaging root causes in prompt/model/retrieval/tool/orchestration layers.

Responsibilities

Design and maintain end-to-end business scenarios for multi-step agentic workflows; evaluate agent outputs for correctness, relevance, groundedness, consistency, tone, and completeness; build and maintain eval datasets and automated regression suites; identify cross-agent, orchestration, and integration issues; test for hallucination, prompt injection, jailbreaks, and data leakage; detect regressions from model upgrades or prompt changes; triage and root-cause issues with Engineering and Product; re-test fixes and validate end-user experience; document test strategy and quality trends.

Seniority

Senior, hands-on IC