QA Engineer
Core
Developing testing strategies, evaluation frameworks, and quality metrics specifically for LLM-powered applications to ensure reliability, accuracy, and trustworthiness.
Role type
AI QA Engineer (GenAI)
Builds
GenAI features including conversational AI, agentic systems, and LLM-powered workflows
Domain
Enterprise SaaS / Generative AI
Deliverable
production ML models
Required skills
QA/test engineering, GenAI technologies (LLMs, prompt engineering), test automation (Python, JavaScript, Selenium, Pytest), API testing (Postman, REST Assured), CI/CD integration, non-deterministic system testing, adversarial testing
Preferred skills
Conversational AI/chatbot testing, ML model evaluation metrics, LLM evaluation frameworks (LangSmith, PromptFoo, Ragas), performance/load testing AI APIs, responsible AI principles, test management tools (TestRail, Zephyr, Jira), Python scripting with LLM APIs
Technologies
Python, JavaScript, Selenium, Pytest, Postman, REST Assured, LangSmith, PromptFoo, Ragas, TestRail, Zephyr, Jira
Responsibilities
Design testing strategies for GenAI features; Develop automated test suites for prompt testing; Create evaluation frameworks for GenAI quality; Build and maintain test datasets and golden examples; Implement monitoring and alerting for quality degradation; Perform adversarial testing; Collaborate on acceptance criteria and quality gates; Develop testing tools for engineers; Conduct user acceptance testing; Document testing procedures and metrics; Partner with Product and Design teams
Seniority
Mid-level, hands-on IC