Machine Learning Engineer, Inference & Serving (Speech LLM)
Core
Build and deploy high-throughput, ultra-low-latency inference systems for large language models and foundational speech models.
Role type
Senior IC machine learning engineer (inference & serving)
Builds
Real-time conversational AI systems with native speech understanding and generation
Domain
AI/ML, Speech Technology, Distributed Systems
Deliverable
production ML models
Required skills
GPU architecture optimization, KV cache management, continuous batching, model quantization, distributed inference pipelines, Kubernetes
Preferred skills
vLLM, TensorRT-LLM, SGLang, NVIDIA Triton, WebRTC, speculative decoding, chunked prefill, multi-GPU tensor parallelism
Technologies
NVIDIA Ampere/Hopper, Kubernetes, WebSockets, FP8, INT8, AWQ, GPTQ
Responsibilities
Optimize inference engines for latency and throughput, manage stateful connections for real-time streaming, deploy multi-node inference pipelines, implement advanced generation algorithms
Seniority
Senior, hands-on IC