Principal QA Engineer – GenAI Testing - Manager
Core
Designing and executing test plans for Large Language Models (LLMs) and AI-driven agents to ensure reliability, safety compliance, and accuracy within financial services.
Role type
Principal QA Engineer (GenAI/LLM Testing)
Builds
Automated test suites and validation frameworks for LLMs and AI agents
Domain
Financial Services / Artificial Intelligence / Large Language Models
Deliverable
production ML models
Required skills
LLM testing, Agents testing, Prompt validation, Red teaming, Safety validation, Automated testing, BDD & TDD, CI/CD
Preferred skills
AI governance, Bias detection, Ethical compliance testing, LLM evaluation frameworks
Technologies
Selenium, Python, Pytest, SoapUI, Jira, Git, Jenkins, Postman, JMeter, OpenAI Evals, Ragas
Responsibilities
Analyze business and technical specifications for AI initiatives. Write functional test cases and perform regression testing. Report bugs, verify fixes, and maintain the bug tracking system. Maintain and execute automated tests using Selenium with Python. Create automated test scripts using the Pytest framework. Automate web services using SoapUI and/or Python. Design and execute test plans for Large Language Models (LLMs), including prompt validation, accuracy checks, and safety compliance. Perform Agents testing for AI-driven workflows to ensure reliability, robustness, and integration quality.
Seniority
Senior, hands-on IC
