CareerPlan

Machine Learning Engineer, Inference & Serving (Speech LLM)

San Francisco, CAFull-time$170k–$320k2026-11-16
Hybrid

Core

Build and deploy high-throughput, ultra-low-latency inference systems for large language models and foundational speech models.

Role type

Senior IC machine learning engineer (inference & serving)

Builds

Real-time conversational AI systems with native speech understanding and generation

Domain

AI/ML, Speech Technology, Distributed Systems

Deliverable

production ML models

Required skills

GPU architecture optimization, KV cache management, continuous batching, model quantization, distributed inference pipelines, Kubernetes

Preferred skills

vLLM, TensorRT-LLM, SGLang, NVIDIA Triton, WebRTC, speculative decoding, chunked prefill, multi-GPU tensor parallelism

Technologies

NVIDIA Ampere/Hopper, Kubernetes, WebSockets, FP8, INT8, AWQ, GPTQ

Responsibilities

Optimize inference engines for latency and throughput, manage stateful connections for real-time streaming, deploy multi-node inference pipelines, implement advanced generation algorithms

Seniority

Senior, hands-on IC