Senior LLM Inference Performance & Evaluation Engineer
Core
Build benchmark pipelines, launch gates, and automated regression detection for LLM inference performance, latency, and cost across model/runtime/provider options.
Role type
Senior IC LLM inference performance & evaluation engineer
Builds
Benchmark pipelines, model launch gates, automated regression detection systems, and routing/fallback recommendations
Domain
AI cloud / LLM inference infrastructure
Deliverable
production ML models
Required skills
LLM serving metrics (TTFT, TPOT/ITL, latency, throughput, concurrency, GPU utilization), Python, benchmark automation, statistical test design, CI/CD integration, Kubernetes
Preferred skills
Go, OpenAI/Anthropic APIs, vLLM/Dynamo/SGLang/Triton-style servers, SLO definition
Technologies
Python, Go, vLLM, Dynamo, SGLang, Triton, Kubernetes, OpenAI API, Anthropic API
Responsibilities
Build benchmark pipelines for TTFT, latency, throughput, and GPU utilization; Create model launch gates for compatibility and behavior; Maintain representative workloads using synthetic and replayed traffic; Compare model/runtime options and publish routing recommendations; Automate regression detection in CI/CD; Work with runtime engineers to identify bottlenecks and with SRE to convert results into SLOs
Seniority
Senior, hands-on IC