QA Engineer (AI Systems)
Core
Designing evaluation frameworks, building golden datasets, and stress-testing non-deterministic AI agent systems for heavy industries.
Role type
Senior IC QA Engineer (AI Systems)
Builds
Automated evaluation pipelines, golden datasets, and quality metrics for LLM-based agents.
Domain
AI/ML, Heavy Industry, Autonomous Systems
Deliverable
production ML models
Required skills
QA/SDET strategy for complex systems, LLM agent testing, Python scripting, eval framework development, dataset design, adversarial testing, root cause analysis, systems thinking
Preferred skills
Human-in-the-loop evaluation, ML/data science background, red-teaming experience, observability tooling, OT environment testing
Technologies
promptfoo, DeepEval, RAGAS, LangSmith, Python
Responsibilities
Design evaluation harnesses for LLM agents, develop golden datasets, define quality metrics beyond accuracy, build automated CI/CD eval pipelines, conduct red-teaming and adversarial testing, test full agent action loops, triage failures across model/prompt/tool/orchestration, establish quality sign-off criteria, mentor engineers on probabilistic testing strategies
Seniority
Senior, hands-on IC