AI Test Engineer - VLabs
Core
Validating the performance, reliability, and integrity of AI-enabled solutions (LLMs, OCR, document classification) in production environments for global tax and immigration service lines.
Role type
Senior IC AI Test Engineer (Quality Engineering)
Builds
Reusable Python-based evaluation frameworks, drift detection systems, ground truth datasets, and AI-as-Judge pipelines embedded in CI/CD.
Domain
AI/ML Quality Assurance, Tax & Immigration Services
Deliverable
production ML models
Required skills
Python, LLM evaluation (deepeval, RAGAS, promptfoo), drift detection, OCR/VLM testing, SQL, API testing (Postman/Newman), ground truth dataset creation
Preferred skills
LangChain, LlamaIndex, Azure AI Foundry, AWS Bedrock, regulated environment experience
Technologies
Python, Postman, Newman, SQL, deepeval, RAGAS, promptfoo, LangChain, LlamaIndex, Azure AI Foundry, AWS Bedrock
Responsibilities
Design adversarial test inputs for hallucination detection; Build and maintain ground truth datasets; Implement AI-as-Judge frameworks for scoring; Operate drift detection frameworks to gate releases; Test end-to-end agentic workflows; Translate evaluation results into actionable insights for stakeholders
Seniority
Senior, hands-on IC