Senior Software Engineer, Inference
Core
Design and implement key components of the LLM inference runtime (engine integration, batching, KV cache management, distributed execution) to achieve low tail latency and high GPU utilization on customer-owned hardware.
Role type
Senior Software Engineer (LLM Inference Runtime)
Builds
Model runtime for HPE AI Essentials, an inference platform for operating large language models on customer-owned infrastructure.
Domain
AI/ML Infrastructure, Cloud Computing, High-Performance Computing
Deliverable
production ML models
Required skills
LLM inference engines (vLLM, SGLang, TensorRT-LLM, TGI), continuous batching, KV cache management, tensor/pipeline parallelism, Kubernetes operators/controllers, Go, Python, C++/CUDA profiling
Preferred skills
Upstream contributions to inference runtimes, disaggregated prefill/decode, RDMA/GPUDirect Storage, MIG/fractional GPU allocation
Technologies
Kubernetes, vLLM, SGLang, TensorRT-LLM, NVIDIA NIM, RDMA, InfiniBand, RoCE
Responsibilities
Design and implement LLM serving deployment components; Partner with teams to improve time-to-first-token and throughput; Build distributed execution capabilities; Evaluate emerging runtimes and quantization schemes; Contribute to orchestration layer (scheduling, autoscaling); Triage and resolve customer issues; Provide code reviews and mentor team members
Seniority
Senior, hands-on IC
