AI Quality Engineer
Core
Build confidence in the quality, safety, and reliability of AI-powered products by defining evaluation criteria, designing tests, and analyzing system behavior.
Role type
AI Quality Engineer (hands-on IC)
Builds
Evaluation datasets, automated evaluators, regression tests, and baselines for conversational AI and autonomous agents.
Domain
Artificial Intelligence / Quality Assurance
Deliverable
production ML models
Required skills
Python, quality engineering, test automation, data science, statistical analysis, root cause analysis, LLM behavior understanding, RAG, prompt injection detection, automated testing frameworks
Preferred skills
Production AI monitoring, JSON/log analysis, Git, modular code, configuration management
Technologies
Python, LLM-as-a-Judge, RAG, Git
Responsibilities
Translate business expectations into measurable evaluation criteria; Design evaluation approaches for non-deterministic AI systems; Build and maintain representative evaluation datasets; Assess dataset quality and detect bias or leakage; Analyze errors by category and root cause; Validate automated evaluators against human judgement; Create baselines and regression tests for system comparisons.
Seniority
Mid-to-Senior, hands-on IC
