Software Engineer, Agent Optimization & Evaluation
Core
Lead the agent optimization and evaluation platform to measure and improve AI agents' impact on engineering velocity, including building evaluation environments, orchestration systems, and feedback loops.
Role type
Senior IC software engineer (AI agent optimization & evaluation)
Builds
Agent optimization harness, evaluation environments, model orchestration systems, and feedback loops for AI agents.
Domain
AI/LLM engineering, software development lifecycle, distributed systems
Deliverable
production ML models
Required skills
Large-scale production system design, AI/LLM and agent architecture, evaluation and benchmarking systems, model routing and orchestration, experimentation frameworks, software engineering fundamentals
Preferred skills
Experience with synthetic task generation, offline replay, human-in-the-loop feedback, regression detection
Technologies
LLMs, AI agents, distributed systems, experimentation frameworks
Responsibilities
Design and build agent optimization harness for evaluating and benchmarking AI agents; Develop Reddit-specific evaluation environments; Build systems for offline replay, automated evaluation, and synthetic task generation; Design model routing and orchestration strategies; Define metrics and experimentation frameworks; Build tooling to analyze agent behavior and investigate failures
Seniority
Senior, hands-on IC
