QA Lead
Core
Lead QA strategy and evaluation for agentic AI product features, transforming ambiguous business requirements into robust scoring systems for brand metrics.
Role type
Senior QA Lead & AI Brand Evaluation Strategist
Builds
Evaluation frameworks, data-driven testing pipelines, and validation tools for non-deterministic AI outputs.
Domain
AI/ML product quality, brand analytics, agentic workflows
Deliverable
production ML models | product features
Required skills
Strategic QA leadership, translating business objectives to evaluation criteria, data-driven statistical analysis, high-volume data literacy, traditional and modern testing methodologies, scripting (Python/JavaScript/SQL), conceptual AI fluency (LLM behaviors, hallucinations)
Preferred skills
LLM evaluation frameworks (Ragas, LangSmith, TruLens), cloud data warehouses (BigQuery, Databricks), custom tool development
Responsibilities
Own end-to-end testing strategy for multi-agent architectures, define brand scoring criteria and rubrics, curate golden test sets, design adversarial testing scenarios, build proprietary QA automation tools, analyze defect trends for root-cause resolution, monitor data observability dashboards
Seniority
Senior, hands-on IC with leadership