Lead AI Tester / QA Lead (LLM & GenAI)
Core
Define and own the testing strategy, governance, and delegation for AI/LLM use cases in a financial services consulting context.
Role type
Lead AI Tester / QA Lead (LLM & GenAI)
Builds
Test strategies, playbooks, CI/CD integration workflows, and evaluation metrics for AI systems.
Domain
Financial services / Artificial Intelligence / Large Language Models
Deliverable
production ML models
Required skills
Test strategy definition, automation in complex environments, delegation and coordination, CI/CD pipeline integration, stakeholder engagement
Preferred skills
AI/ML/LLM system testing, LLM output evaluation (semantic similarity, LLM-as-a-judge), concepts of hallucinations/safety/drift, consulting/client-facing experience
Technologies
Confluence, Jira
Responsibilities
Create and maintain test strategy and playbook for AI solutions; Translate strategy into documentation and workflows; Delegate and coordinate execution across QA engineers and developers; Define AI testing integration into CI/CD pipeline; Establish and track evaluation metrics and monitoring; Guide and engage stakeholders on testing aspects
Seniority
Senior, hands-on IC with strategy focus
