Senior AI Quality & Evaluation Engineer
Core
Build independent proof of retrieval, permissions, and data freshness for RAG systems to serve as acceptance gates for client projects.
Role type
Senior AI Quality & Evaluation Engineer
Builds
Automated test harnesses, golden question suites, negative entitlement tests, and drift alarms for RAG/search systems.
Domain
Enterprise AI consulting, RAG, Search, Security
Deliverable
production ML models
Required skills
Python, Test Automation, RAG evaluation, Security testing (authorization/entitlement), AWS (Bedrock, OpenSearch, CloudWatch), CI/CD scheduling, Root cause analysis
Preferred skills
AWS Certified AI Practitioner, LLM-as-judge scoring, Contact center domain, SOC 2/ISO 27001 evidence practices
Technologies
Python, AWS, Bedrock, OpenSearch, CloudWatch
Responsibilities
Build golden question suites with acceptance thresholds; Build negative entitlement test sets; Build freshness and retirement tests; Automate the evaluation harness; Define QA gate content for analysts; Run pre-pilot evaluation suites and report misses; Write evaluation runbooks and train client stewards
Seniority
Senior, hands-on IC