CareerPlan

Senior Inference Runtime Engineer

Singapore, SGFull-time2026-11-04 → 2026-11-16

Core

Own the performance-critical serving layer of the MaaS platform to make self-hosted LLMs faster, cheaper, and more stable by optimizing the runtime stack behind OpenAI- and Anthropic-compatible APIs.

Role type

Senior Inference Runtime Engineer

Builds

Self-hosted LLM serving infrastructure optimized for latency, throughput, and cost efficiency

Domain

AI Cloud / Large Language Model Inference

Deliverable

production ML models

Required skills

LLM serving runtime optimization, GPU memory management, CUDA/NCCL, continuous batching, speculative decoding, KV cache management, distributed inference, Go, Python, vLLM, Dynamo, SGLang, TensorRT-LLM, TGI, Triton

Preferred skills

None stated

Technologies

vLLM, Dynamo, SGLang, TensorRT-LLM, TGI, Triton, CUDA, NCCL, Go, Python

Responsibilities

Optimize prefill/decode scheduling, continuous batching, KV cache behavior, speculative decoding, long-context serving, and streaming smoothness; Tune and operate vLLM/Dynamo/SGLang/TensorRT-LLM-style runtimes for model-specific latency, throughput, GPU utilization, and cost efficiency; Profile bottlenecks across GPU memory, HBM bandwidth, NCCL/network, tokenizer, frontend/proxy, and model worker paths; Lead high-value model onboarding, including runtime selection, tensor/pipeline parallelism, quantization, context length, and rollback strategy; Define runtime playbooks and safe defaults for reasoning, tool calling, multimodal, prompt cache, and provider-specific parameters; Partner with SRE and performance/evaluation engineers to turn benchmark findings into production runtime improvements

Seniority

Senior, hands-on IC