CareerPlan

Lead AI QA Engineer

Bengaluru, KarnatakaFull-time2026-06-01 → 2026-11-19

Core

Design and build evaluation harnesses, automated test suites, and quality guardrails for production-grade agentic AI systems and LLM-powered features.

Role type

Lead AI QA Engineer

Builds

Internal agent platform on Dataiku, LLM-native analyst workflows, and loyalty program data insights.

Domain

Loyalty technology, Generative AI, Data Engineering

Deliverable

production ML models

Required skills

Python, SQL, LLM testing (prompt regression, tool/function-call validation, RAG correctness, structured-output schema checks), ETL, PyTest, API testing, CI/CD automation, observability instrumentation

Preferred skills

Experience with Dataiku, Snowflake, LangSmith, Langfuse, MLflow, OpenTelemetry, Pydantic, JSON schema validation

Technologies

Python, SQL, Dataiku, Snowflake, LangSmith, Langfuse, MLflow, OpenTelemetry, PyTest, Jenkins, GitHub Actions, Postman, Docker, Git

Responsibilities

Design evaluation harnesses with golden datasets and LLM-as-judge graders; Author automated test suites for prompts, tools, and agent workflows; Validate guardrails for tool execution and hallucination mitigation; Wire evaluations into CI pipelines; Build observability into testing; Own quality end-to-end including release criteria; Partner with data engineers on retrieval testing patterns.

Seniority

Lead, hands-on IC with strategic ownership