Agentic AI Systems Evaluation Engineer
Core
Evaluate AI software to ensure it delivers designed customer experiences and system performance by connecting user stories to measurable outcomes.
Role type
Senior IC systems evaluation engineer (AI inference)
Builds
Benchmarking frameworks, evaluation tools, and actionable feedback for AI workloads
Domain
PC systems and AI inference
Deliverable
production ML models
Required skills
AI inference workloads, benchmarking methodologies, system-level performance, Python/C++ programming, model formats (ONNX/GGUF), telemetry analysis
Preferred skills
Windows/Linux PC evaluation, attention mechanisms/KV caching/quantization, heterogeneous hardware (CPU/GPU/NPU), statistical analysis
Technologies
Python, C++, ONNX, GGUF
Responsibilities
Translate user stories into measurable success criteria and evaluation scenarios; Design repeatable benchmarking frameworks for AI workloads; Compare model formats, runners, and hardware configurations; Analyze inference behavior (latency, token rate, memory); Evaluate system responsiveness under concurrent workloads; Interpret benchmark data to identify meaningful improvements or regressions; Build automation tools for reproducible evaluations
Seniority
Senior, hands-on IC