QA Automation Engineer - AI
Core
Develop testing strategies, evaluation frameworks, and quality metrics for GenAI/Agentic systems and LLM-powered applications.
Role type
QA Automation Engineer (GenAI/LLM)
Builds
Automated test suites, evaluation frameworks, and quality metrics for AI features.
Domain
Artificial Intelligence / Generative AI / Software Quality Assurance
Deliverable
production ML models | product features
Required skills
Python, test automation frameworks (Pytest, API, UI), GenAI technologies (LLMs, prompt engineering), API testing tools (Postman, REST Assured), CI/CD integration, adversarial testing, dataset curation
Preferred skills
Experience with non-deterministic systems, user acceptance testing support
Technologies
Python, Pytest, Postman, REST Assured, CI/CD pipelines
Responsibilities
Develop and execute end-to-end automated tests for evaluations, APIs, UIs, and performance; Implement testing strategies for conversational AI and agentic systems; Run automated test suites for prompt testing and regression; Support evaluation frameworks for accuracy, relevance, safety, and latency; Curate high-quality test datasets and golden examples; Assist in adversarial testing to identify hallucinations or biases; Document testing procedures and known issues.
Seniority
Mid-level, hands-on IC