AI Test Engineer - VLabs
Core
Validating the performance, reliability, and integrity of AI-enabled solutions (LLMs, OCR, document classification, agentic workflows) in production environments to ensure they meet business performance thresholds.
Role type
Senior IC AI Test Engineer (Quality Engineering)
Builds
Reusable Python-based evaluation frameworks, drift detection systems, ground truth datasets, and automated test pipelines embedded in CI/CD.
Domain
AI/ML Quality Assurance, Document AI, LLM Evaluation
Deliverable
production ML models
Required skills
Python, LLM evaluation tools (deepeval, RAGAS, promptfoo), drift detection, SQL, API testing, ground truth dataset creation, agentic workflow testing
Preferred skills
Cloud AI platforms (Azure AI Foundry, AWS Bedrock), VLM testing, regulated environment experience
Technologies
Python, Postman/Newman, SQL, LangChain, LlamaIndex, Azure AI Foundry, AWS Bedrock, deepeval, RAGAS, promptfoo
Responsibilities
Design adversarial test inputs to expose hallucination and failure modes; Build and maintain ground truth datasets; Implement AI-as-Judge frameworks for scoring and calibration; Operate drift detection frameworks to identify performance degradation; Test end-to-end agentic workflows and API endpoints; Translate evaluation results into actionable insights for stakeholders.
Seniority
Senior, hands-on IC