Engineer II - QA Engineer
Core
Developing testing strategies, evaluation frameworks, and quality metrics specifically for GenAI and LLM-powered applications to ensure reliability and trustworthiness.
Role type
GenAI Quality Assurance Engineer
Builds
Testing strategies, automated test suites, evaluation frameworks, and monitoring systems for AI features
Domain
Artificial Intelligence / Generative AI / Enterprise SaaS
Deliverable
production ML models
Required skills
GenAI testing strategies, LLM evaluation frameworks, automated test suite development, adversarial testing, API testing, CI/CD integration, Python scripting, UI testing, performance testing
Preferred skills
Conversational AI testing, ML model evaluation metrics, LLM evaluation tools (LangSmith, PromptFoo, Ragas), responsible AI principles, test management tools
Technologies
Python, JavaScript, Playwright, Puppeteer, Postman, REST Assured, Jira, TestRail, Zephyr
Responsibilities
Design testing strategies for conversational AI and agentic systems; Develop automated test suites for prompt testing and regression; Create evaluation frameworks for accuracy, relevance, safety, and consistency; Build test datasets and golden examples; Implement monitoring for quality degradation; Perform adversarial testing for hallucinations and biases; Collaborate on acceptance criteria; Develop testing tools for engineers; Conduct user acceptance testing; Document testing procedures and metrics.
Seniority
Mid-level, hands-on IC