ML Infrastructure Engineer - New Grad
Core
Build and operate the inference stack for SambaNova's RDU accelerators, serving LLMs with features like speculative decoding and prompt caching.
Role type
new grad ML infrastructure engineer (LLM serving)
Builds
production inference serving stack, public APIs, and accuracy testing for RDU-based AI models
Domain
AI infrastructure, LLM serving, dataflow architecture
Deliverable
production ML models
Required skills
Python, distributed systems, algorithms, data structures, concurrency, systems design, LLM serving frameworks (vLLM, SGLang, TensorRT-LLM), inference techniques (speculative decoding, KV caching, continuous batching), PyTorch
Preferred skills
API/backend service development, ML systems/backend infrastructure, disaggregated systems (prefill/decode), open source contributions
Technologies
Python, vLLM, SGLang, TensorRT-LLM, PyTorch, RDU, SambaRack
Responsibilities
Implement inference features (speculative decoding, constrained decoding, tool calling, prompt caching, long-context inference), adapt vLLM to RDU architecture, build/maintain public inference API endpoints, write accuracy and regression tests, debug latency/throughput/reliability issues
Seniority
Junior, hands-on IC
