Senior QA Automation Engineer - AI
Core
Designing and architecting quality assurance strategies, evaluation frameworks, and metrics specifically for GenAI/Agentic systems and LLM-powered applications.
Role type
Senior SDET for AI (GenAI/LLM)
Builds
Automated test suites, prompt testing systems, regression suites, and real-time monitoring tools for AI features.
Domain
Artificial Intelligence / Generative AI / Software Quality Assurance
Deliverable
production ML models | product features
Required skills
GenAI technologies (LLMs, prompt engineering), Python, test automation frameworks (Pytest, Selenium), API testing (Postman, REST Assured), CI/CD integration, adversarial testing (red-teaming), non-deterministic system testing
Preferred skills
None stated
Technologies
Python, JavaScript, Selenium, Pytest, Postman, REST Assured
Responsibilities
Design comprehensive test plans and evaluation frameworks for LLM applications; Develop end-to-end automated test suites covering evaluations, API, UI, and performance; Create automated prompt testing and regression systems; Establish multi-dimensional evaluation criteria for GenAI quality; Perform adversarial testing to identify hallucinations and biases; Develop internal tools for engineering teams to validate GenAI implementations; Deploy real-time monitoring and alerting for production quality degradation.
Seniority
Senior, hands-on IC
