Senior AI Evals Engineer
Core
Design, develop, and execute robust evaluation frameworks and test strategies for AI agents built using Google ADK or similar agentic frameworks, ensuring safety, ethics, and high-quality performance.
Role type
Senior IC AI Evaluation Engineer (Agentic AI)
Builds
Scalable evaluation pipelines, synthetic test environments, golden datasets, and centralized eval-as-a-service platforms for LLM-based agents.
Domain
Generative AI / Large Language Models (LLMs) / Agentic Workflows / Enterprise Data Security
Deliverable
production ML models
Required skills
LLM evaluation frameworks (DeepEvals, LangSmith, Promptfoo, RAGAS), agentic AI patterns (tool use, ReAct/CoT, RAG), Python scripting, CI/CD integration, statistical literacy, prompt injection testing, data analysis (Pandas, SQL), Kubernetes, observability (Datadog, Grafana)
Preferred skills
Deep Learning/NLP fundamentals, AI ethics and policy background, GIGO (Garbage-in/Garbage-out) testing tools, complex API testing, AWS/GCP certifications
Technologies
Python, TypeScript, Java, Selenium, Playwright, LangGraph, Claude Agent SDK, Jenkins, GitHub Actions, Vertex AI, Gemini, Anthropic Claude, OpenAI, Azure OpenAI, Pandas, SQL, Datadog, Grafana, Kubernetes
Responsibilities
Design end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion; build synthetic testing environments to stress-test agents; curate domain-specific evaluation sets; establish eval-as-a-service platforms; automate regression and drift detection in CI/CD pipelines; define eval taxonomies and rubrics; instrument production observability for agent quality; benchmark and select models; analyze evaluation results for stakeholders; define comprehensive QA strategies; develop and maintain test artifacts; track defects and alignment failures; evaluate new technologies and run proofs-of-concept; partner with policy and legal teams for regulatory compliance; provide technical leadership and mentoring.
Seniority
Senior, hands-on IC with leadership responsibilities