CareerPlan

Senior AI Quality & Evaluation Engineer

USAFull-time2026-11-13 → 2026-11-19

Core

Build independent proof of retrieval, permissions, and data freshness for RAG systems to serve as acceptance gates for client projects.

Role type

Senior AI Quality & Evaluation Engineer

Builds

Automated test harnesses, golden question suites, negative entitlement tests, and drift alarms for RAG/search systems.

Domain

Enterprise AI consulting, RAG, Search, Security

Deliverable

production ML models

Required skills

Python, Test Automation, RAG evaluation, Security testing (authorization/entitlement), AWS (Bedrock, OpenSearch, CloudWatch), CI/CD scheduling, Root cause analysis

Preferred skills

AWS Certified AI Practitioner, LLM-as-judge scoring, Contact center domain, SOC 2/ISO 27001 evidence practices

Technologies

Python, AWS, Bedrock, OpenSearch, CloudWatch

Responsibilities

Build golden question suites with acceptance thresholds; Build negative entitlement test sets; Build freshness and retirement tests; Automate the evaluation harness; Define QA gate content for analysts; Run pre-pilot evaluation suites and report misses; Write evaluation runbooks and train client stewards

Seniority

Senior, hands-on IC