Senior QA & Evaluation Engineer
Core
Design and maintain automated test suites for software and AI systems, integrating quality gates into CI/CD pipelines.
Role type
Senior QA & Evaluation Engineer (AI/ML focus)
Builds
Automated testing frameworks, AI evaluation rubrics, and held-out evaluation datasets
Domain
Software Quality Assurance and AI/LLM Evaluation
Deliverable
production ML models | product features
Required skills
Test automation, .NET, API testing, CI/CD integration, probabilistic AI testing strategies, threshold-based evaluation design, test data management
Preferred skills
LLM evaluation (rubrics, LLM-as-a-judge), information extraction testing, regulated-industry QA, AI/ML testing methodologies
Technologies
.NET, Azure AI Foundry Evaluators, promptfoo, Ragas, DeepEval
Responsibilities
Design and maintain automated test suites covering unit, integration, API, and end-to-end testing; Integrate automated testing into CI/CD pipelines; Build evaluation frameworks for AI capabilities including scored rubrics and LLM-as-a-judge approaches; Measure AI behavior against ground truth and report defect escape metrics
Seniority
Senior, hands-on IC