Agentic QA Engineer, Generative AI Systems
Core
Define and lead QA strategy for agentic and multi-agent AI systems, focusing on reliability, resilience, accuracy, latency, orchestration, and scale.
Role type
Senior IC QA Engineer (Generative AI/Agentic Systems)
Builds
Production-grade generative AI and multi-agent systems
Domain
Generative AI, Multi-agent Systems, Distributed Systems
Deliverable
production ML models
Required skills
Python, LLM evaluation methods, distributed systems testing, chaos engineering, orchestration frameworks, CI/CD, observability, privacy/security compliance
Preferred skills
Multi-agent simulators, MLOps, cloud environments, cost/latency/SLA ownership
Technologies
AWS, Azure, CI/CD, Datadog, GitHub, Grafana, OpenTelemetry, Prometheus, Python, LangChain, LangGraph, LlamaIndex, DSPy
Responsibilities
Define and lead QA strategy for agentic and multi-agent AI systems; Mentor QA engineers and establish testing standards; Test agent orchestration, tool calls, and inter-agent coordination; Develop scenario-fuzzing tests using adversarial inputs; Build resilience test suites covering chaos experiments; Establish latency SLOs and measure end-to-end response times; Create macro-validation frameworks to assess outcomes across multi-step agent workflows; Design load and stress tests for multi-agent graphs; Integrate tests into CI/CD and production monitoring; Define release criteria and conduct operational-readiness assessments.
Seniority
Senior, hands-on IC
