Staff Software Engineer, LLM Inference Platform
Core
Technical lead for end-to-end LLM inference platform performance, optimizing token throughput and cost for models ranging from hundreds of billions to trillions of parameters.
Role type
Staff Software Engineer (LLM Inference Platform)
Builds
High-performance inference engines and disaggregated inference systems for large-scale LLMs.
Domain
Cloud Infrastructure / Large Language Model Systems
Deliverable
production ML models
Required skills
Transformer inference architecture, CUDA kernel tuning, vLLM/SGLang/TensorRT-LLM optimization, MoE parallelism, KV cache management, distributed systems design, performance benchmarking
Preferred skills
Merged PRs in vLLM, cluster-scale GPU operations
Technologies
CUDA, ROCm, HIP, vLLM, SGLang, TensorRT-LLM, NIXL, RDMA, UCX, DeepSeek, Kimi, GLM, Qwen, MiniMax
Responsibilities
Establish unified token performance metrics and load-testing benchmarks; optimize inference engine source code for scheduling and kernel execution; design Prefill-Decode disaggregation and KV transfer paths; design hybrid parallelism strategies for MoE models; write and tune critical CUDA operators and quantization evaluation pipelines.
Seniority
Staff, hands-on IC with strategic ownership