SDET / QA Engineer (+AI evaluation)
Core
QA Engineer / SDET responsible for evaluating, validating, and improving quality across teams, with a specific focus on AI agent evaluation and automated testing.
Role type
Senior IC QA Engineer / SDET (AI evaluation focus)
Builds
Python LLM evaluation tools, diagnostic suites, golden datasets, and automated regression tests for AI agents and core product flows.
Domain
Artificial Intelligence / Machine Learning / Software Quality Assurance
Deliverable
production ML models
Required skills
Python, LLM evaluation, automated testing, Playwright, SonarQube, exploratory testing, test strategy design, stakeholder liaison, quality metrics analysis
Preferred skills
AI agent evaluation, multi-turn conversational testing, regression testing frameworks
Technologies
Python, Playwright, SonarQube, LLMs
Responsibilities
Take ownership of a Python LLM evaluation tool and diagnostic suite; Refine scoring criteria and design multi-turn conversational evaluations for AI agents; Set up automated regression testing for core product flows using Playwright; Implement SonarQube quality gate monitoring and daily deviation reporting; Partner with senior stakeholders on QA strategy and standards; Estimate, prioritize, plan, and coordinate testing activities.
Seniority
Senior, hands-on IC