Agentic AI Test Engineer
Core
Build automated evaluation loops for AI Agents using LLM-as-a-judge patterns and maintain robust Web and API test suites for Comcast's Xfinity Assistant.
Role type
Senior SDET specializing in Agentic AI and Quality Engineering
Builds
Automated evaluation frameworks, test utilities, and CI/CD pipelines for AI-driven chat systems
Domain
AI/LLM evaluation, Quality Engineering, Full-Stack Automation
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Python (expert), LLM-as-a-Judge implementation, Web automation, API automation, CI/CD pipeline integration, Root cause analysis, Code review
Preferred skills
TypeScript, Java, React, Streamlit, Jenkins, Concourse
Technologies
Python, TypeScript, Java, React, Streamlit, Jenkins, Concourse, RESTful APIs
Responsibilities
Design and implement automated evaluation frameworks using LLMs; Perform hands-on validation of AI Agents in complex environments; Automate testing of real-time data streams and LLM outputs; Develop and maintain automation across TypeScript, Java, and React environments; Integrate automated tests into CI/CD pipelines; Participate in technical code reviews and perform root cause analysis on automation failures
Seniority
Senior, hands-on IC
