Quality Assurance Engineer – AI Agentic Applications, AVP
Core
Design, build, and execute test strategies for AI agentic applications, validating LLM accuracy, RAG performance, agent planning, and safety.
Role type
Senior IC Quality Assurance Engineer (AI/GenAI)
Builds
Automated test suites and quality gates for AI-powered agentic workflows
Domain
Financial services + Generative AI
Deliverable
production ML models
Required skills
Test automation framework design, LLM evaluation (hallucination, grounding), RAG testing, AI safety/prompt injection testing, Python or Java, CI/CD integration
Preferred skills
LangChain/LangGraph, vector databases, Kafka, distributed systems testing, financial regulatory knowledge
Technologies
Langsmith, Langfuse, Ragas, LangChain, LlamaIndex, Vertex AI, OpenAI, Kafka, SQL, REST APIs
Responsibilities
Design and maintain automation frameworks for AI agents; Validate RAG ingestion and citation accuracy; Perform AI safety testing for prompt injection and data leakage; Establish quantitative AI evaluation metrics; Analyze production agent failures to create regression tests
Seniority
Senior, hands-on IC