Senior AI QA Test Analyst
Core
Designing and implementing automated testing frameworks for sophisticated generative AI, language, and multimodal systems to validate quality, coherence, safety, and performance.
Role type
Senior IC Test Automation Specialist (Generative AI)
Builds
Automated test suites and validation frameworks for AI-based products in Investor Relations and Sustainable Lens software.
Domain
FinTech / Generative AI / Machine Learning
Deliverable
production ML models
Required skills
Python, test automation frameworks (RAGAS, Deep eval, TrueLens, AWS Bedrock Evaluator, pytest, Robot Framework), AI/ML algorithm validation, statistical analysis, prompt engineering testing, model versioning testing, bias analysis, vectorized databases, AWS Bedrock, Sonnet/LLM algorithms, performance testing of distributed systems
Preferred skills
Cloud platforms (AWS) for AI deployment, containerization (Docker, Kubernetes), large-scale distributed systems testing
Responsibilities
Develop and maintain automated test suites for generative AI models; Create testing strategies to validate AI-generated content quality and safety; Design automation frameworks for generative AI products; Collaborate with AI researchers and data scientists; Develop tools to generate diverse test data sets and simulate real-world scenarios; Implement automated tests for prompt engineering, fine-tuning, and model versioning; Analyze test results to report on model performance, biases, and improvement areas
Seniority
Mid-Senior, hands-on IC
