Senior QA Engineer
Core
Design and maintain end-to-end business scenarios for AI agents, including routing, tool calls, orchestration, and multi-turn conversations, while evaluating responses for correctness, relevance, and safety.
Role type
Senior QA Engineer (AI Agents & LLMs)
Builds
Evaluation datasets, golden sets, edge cases, adversarial prompts, and automated regression suites for AI agent workflows.
Domain
Artificial Intelligence / Large Language Models / Agentic Workflows
Deliverable
production ML models
Required skills
AI agent evaluation, LLM testing, test automation (Selenium, Playwright, TestComplete), API testing, RAG, multi-agent orchestration, prompt injection testing, hallucination detection, rubric creation, regression suite maintenance.
Preferred skills
Experience with LLM-as-judge methods, understanding of non-deterministic output evaluation.
Responsibilities
Design and maintain end-to-end business scenarios covering agent routing, tool and function calls, orchestration, handoffs, and multi-turn conversations; Evaluate agent responses for correctness, relevance, groundedness, consistency, tone, and completeness using human and automated methods; Build and maintain evaluation datasets, golden sets, edge cases, adversarial prompts, and automated regression suites; Identify cross-agent, orchestration, context, integration, tool-call, looping, and runaway-behavior issues; Test for hallucinations, prompt injection, jailbreaks, data leakage, and unsafe or out-of-scope responses; Detect and quantify regressions caused by model, prompt, retrieval, or knowledge changes; Triage issues with Engineering and Product, determine root causes, re-test fixes, and validate the end-user experience; Document test strategy, evaluation results, and quality trends for global stakeholders.
Seniority
Senior, hands-on IC