Senior QA Automation Engineer (AI & LLM Testing)
Core
Design, develop, and maintain automated test suites for web applications, APIs, and AI-powered features, including evaluating LLM outputs for accuracy and hallucination risks.
Role type
Senior QA Automation Engineer (AI & LLM Testing)
Builds
Automated test suites for web applications, APIs, and AI-powered features
Domain
Software Quality Assurance, Artificial Intelligence, Large Language Models
Deliverable
production ML models | product features
Required skills
Test automation, API testing, LLM quality evaluation, Prompt testing, Playwright, Python scripting, Postman, CI/CD integration, AI coding assistants
Preferred skills
Mobile test automation, iOS/Android platform testing, MCP or agent-assisted testing
Technologies
Playwright, Python, Postman, Promptfoo, GitHub Copilot, Appium, XCUITest
Responsibilities
Design and maintain automated test suites for web applications and APIs; Evaluate LLM outputs for response accuracy, relevance, and hallucination risks; Create prompt tests and quality checks using LLM evaluation frameworks; Build end-to-end UI automation using Playwright; Develop reusable Python scripts for API testing and automation utilities; Create and maintain Postman collections for REST API testing with CI/CD integration
Seniority
Senior, hands-on IC
