QA AI Automation Engineer
Core
Build and maintain evaluation suites, golden datasets, and automated gates for non-deterministic AI systems, LLM agents, and autonomous coding workflows within a regulated wealth management platform.
Role type
Senior IC QA AI Automation Engineer (LLM/Agent Evaluation)
Builds
Evaluation suites, golden datasets, multi-agent CI/CD workflows, and automated quality standards for agent-authored code.
Domain
Financial services / Wealth management / AI & LLMs
Deliverable
production ML models
Required skills
LLM agent patterns, prompt engineering, tool/function calling, orchestration, major model APIs (OpenAI, Anthropic, Azure OpenAI), AI-assisted development tools, Python/TypeScript/Java/Kotlin/C#, evaluation criteria definition, CI/CD pipeline integration, API testing, third-party integration validation.
Preferred skills
Evaluation/observability tools (Langfuse, Promptfoo, OpenTelemetry), RAG fundamentals (embeddings, chunking, vector search), workflow orchestration (n8n), Azure cloud/.NET ecosystem, BDD/Selenium UI automation, regulated industry experience, mentoring distributed QA teams.
Technologies
Azure, C#, CI/CD, Claude Code, Cloud, Copilot, Java, JavaScript, Kotlin, LLM, Langfuse, n8n, OpenTelemetry, Python, RAG, Selenium, TypeScript, ASP.NET
Responsibilities
Build evaluation suites measuring correctness, citation fidelity, retrieval quality, refusal behavior, tone, latency, and cost; curate and version golden datasets and adversarial tests; develop LLM-as-judge and rubric-based scoring; integrate evaluations into CI to measure regressions; design and operate multi-agent CI/CD workflows for defect detection and auto-fixing; define agent roles, handoffs, and trust checks; establish quality standards and automated gates for agent-authored code; detect common agent-generated code failures; collaborate with product/engineering/AI Labs on acceptance criteria; contribute to release-readiness decisions; maintain and modernize API/UI/integration automation.
Seniority
Senior, hands-on IC
