Senior Inference Runtime Engineer
Core
Own the performance-critical serving layer of the MaaS platform to make self-hosted LLMs faster, cheaper, and more stable by optimizing the runtime stack behind OpenAI- and Anthropic-compatible APIs.
Role type
Senior Inference Runtime Engineer
Builds
Self-hosted LLM serving infrastructure optimized for latency, throughput, and cost efficiency
Domain
AI Cloud / Large Language Model Inference
Deliverable
production ML models
Required skills
LLM serving runtime optimization, GPU memory management, CUDA/NCCL, continuous batching, speculative decoding, KV cache management, distributed inference, Go, Python, vLLM, Dynamo, SGLang, TensorRT-LLM, TGI, Triton
Preferred skills
None stated
Technologies
vLLM, Dynamo, SGLang, TensorRT-LLM, TGI, Triton, CUDA, NCCL, Go, Python
Responsibilities
Optimize prefill/decode scheduling, continuous batching, KV cache behavior, speculative decoding, long-context serving, and streaming smoothness; Tune and operate vLLM/Dynamo/SGLang/TensorRT-LLM-style runtimes for model-specific latency, throughput, GPU utilization, and cost efficiency; Profile bottlenecks across GPU memory, HBM bandwidth, NCCL/network, tokenizer, frontend/proxy, and model worker paths; Lead high-value model onboarding, including runtime selection, tensor/pipeline parallelism, quantization, context length, and rollback strategy; Define runtime playbooks and safe defaults for reasoning, tool calling, multimodal, prompt cache, and provider-specific parameters; Partner with SRE and performance/evaluation engineers to turn benchmark findings into production runtime improvements
Seniority
Senior, hands-on IC