Senior ML Engineer | Kimchi (Inference Stack Optimization)
Core
Optimize the inference stack for LLMs to improve throughput, latency, and cost per token by tuning engines, schedulers, and hardware configurations.
Role type
Senior IC machine-learning engineer (inference stack optimization)
Builds
Serverless LLM inference services on GPU fleets (SGLang, vLLM, TensorRT-LLM) for customers
Domain
Cloud infrastructure + Large Language Model serving
Deliverable
production ML models
Required skills
Python (production-grade), vLLM, SGLang, TensorRT-LLM, CUDA, Kubernetes, multi-GPU/multi-node distributed systems, quantization, shared filesystems, performance profiling, benchmarking
Preferred skills
LLM routing, online A/B testing with drift monitoring, conformal methods
Technologies
Python, SGLang, vLLM, TensorRT-LLM, PyTorch, CUDA, Kubernetes, virtiofs, ClickHouse, OpenTelemetry, Prometheus, Grafana, GitLab CI
Responsibilities
Tune continuous batching, chunked prefill, speculative decoding, and scheduler parameters to maximize GPU throughput. Profile and fix bottlenecks in TTFT and inter-token latency (compute, memory, storage, network). Implement hierarchical/paged attention, prefix caching, and quantized KV caches. Optimize weight loaders and JIT caches to reduce cold starts. Manage distributed inference topologies and network-aware placement. Define technical direction and benchmarking strategies for the team.
