Senior GenAI QA Engineer - Toronto
Core
Lead quality assurance and evaluation strategy for a production-grade Agentic AI platform in the healthcare domain, focusing on safety, hallucination risk, and PII/PHI protection.
Role type
Senior GenAI QA Engineer (Agentic AI)
Builds
Automated evaluation tools, test datasets, and quality checks for non-deterministic AI systems and backend services.
Domain
Healthcare + Generative AI / LLMs
Deliverable
production ML models
Required skills
API testing, automation, Python, CI/CD integration, AI quality evaluation, hallucination risk assessment, guardrail evaluation, adversarial/red-team testing
Preferred skills
RAG-based systems, Agentic AI systems, AWS Bedrock, Azure OpenAI, Anthropic Claude, LangGraph, LangChain, Semantic Kernel, Promptfoo, Langfuse, DeepEval
Technologies
Python, AWS, Azure, Langfuse, DeepEval, LangChain, LangGraph
Responsibilities
Lead QA and evaluation strategy for Agentic AI platform; Develop automated evaluation tools in Python; Create repeatable evaluation frameworks and test datasets; Test AI guardrails using functional, adversarial, and red-team approaches; Own core software QA foundation for APIs and backend services.
Seniority
Senior, hands-on IC