Senior AI Test Automation Engineer
Core
Design, develop, and execute automated testing and evaluation solutions for traditional software and LLM-powered applications, operating in a forward-deployed capacity to capture real-world usage patterns and validate LLM behavior.
Role type
Senior IC AI Test Automation Engineer
Builds
Scalable automation and evaluation frameworks, offline evaluation suites, and CI/CD pipeline integrations for LLM applications.
Domain
Artificial Intelligence / Large Language Models (LLM) / Software Quality Assurance
Deliverable
production ML models | product features
Required skills
LLM evaluation and observability, LangSmith, LangGraph, Playwright (TypeScript), Python, TypeScript, API/UI automation, CI/CD integration, nondeterministic testing strategies, heuristic and LLM-as-judge evaluators
Preferred skills
RAG evaluation, guardrails validation, prompt engineering, BDD/Gherkin, contract testing, performance/security test automation, AI-assisted engineering practices
Technologies
LangSmith, LangGraph, Playwright, TypeScript, Python, Azure DevOps, Jenkins, GitHub Actions, Git
Responsibilities
Design and maintain offline evaluation suites for regression and benchmarking; develop and calibrate evaluators including LLM-as-judge and heuristic checks; instrument end-to-end tracing for LangGraph agents; integrate automated tests and eval runs into CI/CD pipelines; analyze agent trajectories to distinguish LLM variance from product defects; manage annotation queues and human-in-the-loop feedback workflows; report on test execution results and quality trends.
Seniority
Senior, hands-on IC
