Senior Product Engineer (Agent Training and Evals)
Core
Build infrastructure to evaluate AI agents by capturing agent traces at scale alongside real human expert work in safe environments.
Role type
Senior IC product engineer (AI agent training and evaluation)
Builds
Scalable systems for agent evaluation, data capture, and benchmarking
Domain
AI/ML infrastructure, Agentic AI, Human-in-the-loop data systems
Deliverable
production ML models
Required skills
Cloud infrastructure, distributed systems, AI model/agent evaluation, system deployment and scaling, security, product mindset
Preferred skills
Image/video/audio processing, Python
Technologies
Google Cloud Platform, Python, JavaScript, TypeScript, FastAPI, Vue.js, Kubernetes, Postgres SQL, GitHub Actions, DataDog
Responsibilities
Build tooling for capturing and processing data from agents and humans at scale, solve problems around compute orchestration and reliability, develop approaches for training and benchmarking AI agents, partner with product and research to deliver working products
Seniority
Senior, hands-on IC