Software Engineer II & Senior Software Engineer - AI Inference and Serving (CoreAI)
Core
Optimize AI inference systems for profile execution, memory management, and dependency bottlenecks to improve throughput, latency, and cost.
Role type
Senior IC software engineer (AI inference and serving)
Builds
Secure, tenant-isolated model-serving runtimes and gateways
Domain
AI/ML infrastructure, high-performance computing
Deliverable
production ML models
Required skills
C++, C#, Rust, Go, Java, Python, networking, concurrency, debugging, containerization, Kubernetes, vLLM, SGLang, TensorRT-LLM
Preferred skills
cache-aware routing, admission control, continuous batching, KV-cache management, load testing, failure testing, observability, automated testing, reproducible benchmarking, near-linear throughput scaling
Responsibilities
Improve cache-aware routing, admission control, continuous batching, or KV-cache management; Validate time-to-first-token, tail latency, and cost per token; Build secure, tenant-isolated systems with automated testing and observability; Participate in operational support and improve services using production feedback; Preserve model quality and numerical correctness; Use reproducible benchmarks to improve throughput, latency, and cost; Pursue near-linear throughput scaling across GPU and node configurations
Seniority
Senior, hands-on IC