Software Engineer, Agent Eval Platform
Core
Build the judgement layer of an agent evaluation platform, including rubrics, judges, calibration against human labels, and orchestration for scoring AI agent trajectories in enterprise systems.
Role type
Senior IC backend/infrastructure engineer (agent evaluation platform)
Builds
Eval orchestration runtime, agent observability/tracing system, and stateful simulation environments for enterprise systems.
Domain
AI Agents / Enterprise Automation / Observability
Deliverable
production ML models
Required skills
Distributed systems, Orchestration and workflow runtimes, Observability internals, Concurrent and async programming, Data-intensive pipelines, gRPC/protobuf service design
Preferred skills
Python, Go, LLMs judging LLMs methodology
Technologies
Python, Go, OpenTelemetry, Temporal, Airflow, Argo, gRPC, protobuf
Responsibilities
Build runtime for multi-turn agent scenarios execution and scoring; Lead migration to OpenTelemetry-native observability; Design stateful simulation environments for enterprise systems.
Seniority
Senior, hands-on IC