QA Engineer
Core
QA Engineer responsible for automated end-to-end coverage, regression suites, and evaluation engineering for non-deterministic LLM/agent outputs in an SAP delivery operating system.
Role type
Senior IC QA Engineer (Test Automation & Evaluation Engineering)
Builds
Automated test suites, evaluation harnesses, golden datasets, and quality gates for an agentic orchestration platform.
Domain
Enterprise Software / SAP Implementation / AI/LLM Systems
Deliverable
production ML models | product features
Required skills
Test automation (Python/TypeScript), E2E frameworks (Playwright/Cypress), API testing, CI/CD pipeline management, LLM/ML output evaluation, severity judgment, AI tooling proficiency, high agency.
Preferred skills
LLM eval design, hallucination detection, enterprise ecosystem knowledge (SAP/Salesforce/ERP), performance/load testing, code repair capabilities.
Technologies
Python, TypeScript, Playwright, Cypress, CI/CD pipelines, LLM evaluation frameworks.
Responsibilities
Build and maintain end-to-end, integration, and regression test coverage; create evaluation harnesses and scoring criteria for agent-generated artifacts; define quality gates and release criteria; ensure CI reliability and fix flaky tests; run exploratory testing on customer-facing workflows; triage defects and drive them to closure; feed quality failure patterns into product roadmap.
Seniority
Senior, hands-on IC
