Agentic QA Engineer
Core
Design and implement testing strategies, automated evaluation frameworks, and quality metrics for AI agents, LLM-powered applications, and agentic workflows to ensure reliability in real-world environments.
Role type
Agentic QA Engineer (SDET)
Builds
Automated test frameworks, evaluation pipelines, and quality checks for AI-driven systems
Domain
Artificial Intelligence / Generative AI / Software Quality Engineering
Deliverable
production ML models
Required skills
Python, JavaScript, TypeScript, Test Automation, API Testing, CI/CD, Debugging, Analytical Thinking, Git
Preferred skills
LLM Testing, Agentic Frameworks (LangGraph, LangChain, CrewAI), LLM Evaluation, Prompt Testing, AI Observability, Cloud Platforms, Distributed Systems
Technologies
Pytest, Playwright, Cypress, Selenium, LangGraph, LangChain, CrewAI
Responsibilities
Design testing strategies for AI agents and agentic workflows; Build automated test frameworks for functional, integration, and end-to-end testing; Develop evaluation pipelines to measure agent accuracy and reliability; Test agent tool usage, API integrations, and multi-step reasoning workflows; Identify and troubleshoot failures like hallucinations and incorrect tool calls; Integrate automated testing into CI/CD pipelines; Collaborate with AI engineers and product teams to improve system reliability.
Seniority
Mid-Senior, hands-on IC