Senior Quality Assurance Engineer
Core
Own and scale testing for AI voice and chat agents, LLM output quality, and RAG pipelines in a regulated BFSI context.
Role type
Senior IC QA Automation Engineer (AI)
Builds
Automated testing frameworks, evaluation harnesses, and CI/CD pipelines for AI pipelines.
Domain
Banking, financial services, and insurance industry; AI/LLM/voice technology.
Deliverable
production ML models
Required skills
LLM evaluation, prompt engineering, RAG testing, voice pipeline testing (STT/TTS), UI/E2E automation, API test automation, CI/CD pipeline construction, database testing, load/performance testing, Git discipline, cloud knowledge, agile sprint lifecycle.
Preferred skills
TypeScript, LiveKit/WebRTC, Azure Speech/Google Cloud Speech/Deepgram/Whisper, GPU inference (vLLM/TGI/Triton), Azure OpenAI, Langfuse/LangSmith/DeepEval/RAGAS, Cypress, Postman, Terraform/Pulumi/CloudFormation, Docker/Kubernetes, PostHog.
Technologies
Playwright, Langfuse, LangSmith, DeepEval, RAGAS, LiveKit, WebRTC, Azure Speech, Google Cloud Speech, Deepgram, Whisper, vLLM, TGI, Triton, Azure OpenAI, Braintrust, Promptfoo, Cypress, Postman, AWS, Azure, GCP, Terraform, Pulumi, CloudFormation, Docker, Kubernetes, PostHog.
Responsibilities
Build and maintain automated testing for AI voice and chat agents; Test LLM output quality using evaluation harnesses; Validate RAG pipelines; Test voice pipelines (STT/TTS accuracy and latency); Automate across UI/E2E, API, and AI output layers; Build CI/CD pipelines; Own release quality and regression strategy; Cover database and load/performance testing; Extend automation to desktop and mobile clients.
Seniority
Senior, hands-on IC
