Agentic Evaluation & Testing Specialist, Kitchener, ON
Core
Build automated test infrastructure and evaluation workflows for an AI-first industrial operations software product, validating both the system and agent-generated outputs (skills, prompts, configurations) for correctness, safety, and performance.
Role type
Senior IC software engineer (test automation & agent evaluation)
Builds
Automated test infrastructure, contract tests, and an internal evaluation workflow for AI agents
Domain
Industrial operations software / AI / Manufacturing
Deliverable
production ML models
Required skills
Test automation, distributed system verification, LLM evaluation, Python, Kubernetes, CI/CD integration, contract testing, API schema validation
Preferred skills
Agent evaluation frameworks, AI observability tooling, property-based testing, safety-critical software testing, industrial systems (MES/SCADA/OPC UA), LLM-as-judge design
Technologies
Python, Kubernetes, Langfuse, LangSmith, Promptfoo
Responsibilities
Build cross-component contract tests and end-to-end tests for a multi-site distributed system; Design evaluation methods for agent behavior beyond simple pass/fail; Integrate security checks into the agent evaluation workflow; Maintain test expectations independent of implementation; Use coding agents for test development and failure analysis
Seniority
Senior, hands-on IC