Software Engineer - Evals for Agents
Core
Design and scale innovative, secure systems for evaluating AI agents, including online/offline evaluation platforms, experimentation tools, and continuous improvement cycles.
Role type
Senior IC backend software engineer (AI agent evaluation)
Builds
Evaluation platforms for AI agents, offline experimentation tools (replay, simulation), and APIs/CLI for developer integrations
Domain
Generative AI, Agent Development, E-commerce, Fintech
Deliverable
production ML models
Required skills
Microservices architecture, Distributed systems, Asynchronous processing, Go, Python, RESTful API design, Relational and NoSQL databases, LLM APIs
Preferred skills
Evaluation frameworks, LLM-as-judge, Observability with LLMs, Agent development platforms, Data analysis, Statistics, SQL, BigQuery, Experimental design
Responsibilities
Evolve and maintain the agent evaluation platform (online and offline), Build offline experimentation tools (conversation replay, user simulation, version comparison), Create continuous improvement cycles for agents, Make the platform simple to adopt via APIs and CLI, Collaborate with product and business teams
Seniority
Senior, hands-on IC
