Agentic AI Quality Engineer
Core
Evaluate autonomous agents, multi-step workflows, and RAG applications for quality, safety, and performance using automated metrics and human-in-the-loop review.
Role type
Senior IC Agentic AI Quality Engineer
Builds
Evaluation frameworks, golden datasets, test harnesses, and production monitoring for AI agents.
Domain
Artificial Intelligence / Agentic AI / MLOps
Deliverable
production ML models
Required skills
Agentic AI evaluation, Python, Prompt engineering, Adversarial testing, Schema validation, LLM-as-a-Judge frameworks, CI/CD integration, Data lineage management
Preferred skills
TMAP or ISTQB Certification, Open-source contributions
Technologies
Python, MLflow, Galileo, PyRIT, Vector databases
Responsibilities
Automate evaluation of multi-turn conversations and long-horizon tasks; Implement safety filters and golden datasets; Run red-team and prompt-injection tests; Embed test harnesses in CI/CD; Translate business risks into measurable quality criteria.
Seniority
Senior, hands-on IC
