CareerPlan

Senior GenAI QA Engineer - Toronto

USAFull-time$216320k–$259584k2026-11-19 → 2026-11-20
Hybrid · Canada

Core

Lead quality assurance and evaluation strategy for a production-grade Agentic AI platform in the healthcare domain, focusing on safety, hallucination risk, and PII/PHI protection.

Role type

Senior GenAI QA Engineer (Agentic AI)

Builds

Automated evaluation tools, test datasets, and quality checks for non-deterministic AI systems and backend services.

Domain

Healthcare + Generative AI / LLMs

Deliverable

production ML models

Required skills

API testing, automation, Python, CI/CD integration, AI quality evaluation, hallucination risk assessment, guardrail evaluation, adversarial/red-team testing

Preferred skills

RAG-based systems, Agentic AI systems, AWS Bedrock, Azure OpenAI, Anthropic Claude, LangGraph, LangChain, Semantic Kernel, Promptfoo, Langfuse, DeepEval

Technologies

Python, AWS, Azure, Langfuse, DeepEval, LangChain, LangGraph

Responsibilities

Lead QA and evaluation strategy for Agentic AI platform; Develop automated evaluation tools in Python; Create repeatable evaluation frameworks and test datasets; Test AI guardrails using functional, adversarial, and red-team approaches; Own core software QA foundation for APIs and backend services.

Seniority

Senior, hands-on IC