QA Automation Engineer
Core
Building automated testing and evaluation frameworks for a KI platform and chatbot to ensure quality before launch.
Role type
QA Automation Engineer (Freelance)
Builds
Automated test suites and LLM/RAG evaluation pipelines for a chatbot and agent platform.
Domain
AI/ML, RAG, Chatbots, Quality Assurance
Deliverable
production ML models | product features
Required skills
Test automation frameworks (Playwright, WebdriverIO), CI/CD integration (GitLab), API testing, LLM/RAG evaluation, Test data management, Agile methodologies
Preferred skills
Azure AI Foundry, Langfuse, RAGAS, DeepEval, Promptfoo, LangSmith, Arize Phoenix, Groundedness/Relevance metrics, Strapi CMS
Technologies
Playwright, WebdriverIO, GitLab CI/CD, Azure AI Foundry, Langfuse, RAGAS, DeepEval, Promptfoo, LangSmith, Arize Phoenix, Strapi
Responsibilities
Build and maintain automated tests for regression-critical frontend, API, and integration flows; Integrate automated tests into GitLab CI/CD pipelines as quality gates; Define automated test cases for chatbot, API, and frontend; Set up automated LLM/RAG evaluations with datasets and quality metrics; Evaluate RAG answers against criteria like factual correctness and source grounding; Structure QA automation and evaluation processes for the platform; Prepare Go/No-Go assessments for launch.
Seniority
Mid-Senior, hands-on IC