Lead AI QA Engineer
Core
Design and build evaluation harnesses, automated test suites, and quality guardrails for production-grade agentic AI systems and LLM-powered features.
Role type
Lead AI QA Engineer
Builds
Internal agent platform on Dataiku, LLM-native analyst workflows, and loyalty program data insights.
Domain
Loyalty technology, Generative AI, Data Engineering
Deliverable
production ML models
Required skills
Python, SQL, LLM testing (prompt regression, tool/function-call validation, RAG correctness, structured-output schema checks), ETL, PyTest, API testing, CI/CD automation, observability instrumentation
Preferred skills
Experience with Dataiku, Snowflake, LangSmith, Langfuse, MLflow, OpenTelemetry, Pydantic, JSON schema validation
Technologies
Python, SQL, Dataiku, Snowflake, LangSmith, Langfuse, MLflow, OpenTelemetry, PyTest, Jenkins, GitHub Actions, Postman, Docker, Git
Responsibilities
Design evaluation harnesses with golden datasets and LLM-as-judge graders; Author automated test suites for prompts, tools, and agent workflows; Validate guardrails for tool execution and hallucination mitigation; Wire evaluations into CI pipelines; Build observability into testing; Own quality end-to-end including release criteria; Partner with data engineers on retrieval testing patterns.
Seniority
Lead, hands-on IC with strategic ownership