Senior Backend Engineer, Inference Platform
Core
Own and scale the inference gateway that routes every model call, managing provider selection, fallback, caching, and cost metering for AI search infrastructure.
Role type
Senior Backend Engineer (Inference Platform)
Builds
Inference gateway, control plane, routing logic, and cost attribution systems for AI model serving.
Domain
AI Infrastructure / Large Language Model Serving
Deliverable
production ML models
Required skills
Model serving (vLLM, SGLang), continuous batching, KV/prefix caching, proxy/gateway architecture, streaming responses, Python, Kubernetes, cost metering, request-path enforcement.
Preferred skills
Experience at inference provider or GPU cloud, open-source serving stack contributions, fine-tuning/evaluating models, usage metering/billing at scale.
Responsibilities
Own the inference gateway including provider/model selection and fallback; build intelligence layer for autonomous model selection; make routing decisions accounting for cache behavior and latency; build cost attribution and metering; build request-path enforcement; ship changes in shadow mode.
Seniority
Senior, hands-on IC