AI QA Evaluation Engineer
Core
Define, execute, and improve testing and evaluation practices for enterprise AI agents and agentic workflows to ensure functional quality, safety, and reliability.
Role type
QA / Evaluation Engineer for Enterprise AI Agents
Builds
Evaluation frameworks, test standards, and validated AI solutions for enterprise deployment
Domain
Enterprise AI, Agentic Workflows, Software Quality Assurance
Deliverable
production ML models
Required skills
Test design, test execution, defect management, structured test approaches for complex systems, agile/Kanban delivery, analytical problem-solving
Preferred skills
AI evaluation frameworks (groundedness, safety, hallucination detection), test automation, CI/CD-integrated testing, prompt testing, enterprise governance
Technologies
Microsoft Copilot Studio, Azure AI Foundry
Responsibilities
Define and implement testing frameworks for AI agents; Develop and execute test plans, cases, and benchmark scenarios; Partner with DevOps to validate solutions in build and deployment phases; Maintain development and test environments; Support user acceptance testing for end-user built agents; Identify and track defects with engineering teams; Collaborate with Security and Legal on compliance and responsible AI requirements
Seniority
Mid-level, hands-on IC