QA AI Automation Engineer
Core
Build evaluation frameworks and multi-agent quality workflows for non-deterministic AI systems in a regulated wealth management platform.
Role type
Senior IC QA AI Automation Engineer
Builds
AI evaluation suites, multi-agent defect detection pipelines, and automated code quality gates for agent-authored code.
Domain
AI/ML engineering, Regulated Wealth Management
Deliverable
production ML models
Required skills
AI evaluation framework design, LLM-as-judge implementation, multi-agent workflow orchestration, CI/CD pipeline integration, Python/TypeScript/Java/Kotlin/C#, API testing, adversarial test set curation, autonomous coding agent quality assurance
Preferred skills
Langfuse, Promptfoo, OpenTelemetry, RAG fundamentals, n8n, Azure cloud services, BDD/Selenium, Financial services domain experience
Technologies
OpenAI, Anthropic, Azure OpenAI, Claude Code, Copilot, Codex, Langfuse, Promptfoo, OpenTelemetry, New Relic, n8n, Azure, Selenium
Responsibilities
Build and maintain evaluation suites for AI assistants and APIs; design LLM-as-judge scoring systems; operate multi-agent workflows in CI/CD to auto-detect and fix defects; define automated gates for agent-authored code quality; partner with product and engineering on acceptance criteria for AI features.
Seniority
Senior, hands-on IC
