QA Automatizador- IA
Core
Ensuring quality and reliability of AI applications and agents through test automation, validation scenario definition, and continuous evaluation of language models (LLMs).
Role type
Senior IC QA Automation Engineer (AI/LLM)
Builds
Robust, scalable AI solutions and LLM-based agents
Domain
Artificial Intelligence, Large Language Models (LLMs), Software Quality Assurance
Deliverable
production ML models
Required skills
AI solution test automation, AI agent validation, Azure DevOps, CI/CD pipelines, LLM evaluation (hallucination, groundedness, relevance), YAML/JSON test scenario design, defect analysis
Preferred skills
Collaborative work with multidisciplinary teams
Technologies
Azure DevOps, YAML, JSON, LLMs
Responsibilities
Design, develop, and maintain test automation frameworks for AI solutions; Execute functional, integration, and AI agent validation tests; Define and manage test scenarios using YAML and JSON; Evaluate LLM performance metrics including hallucination and groundedness; Integrate automated testing processes into Azure DevOps CI/CD pipelines; Analyze test results, document findings, and propose process improvements.
Seniority
Mid-level, hands-on IC