Senior Software Engineer, Private Cloud Compute Inference Routing
Core
Design and operate distributed inference routing and scheduling services for Apple's Private Cloud Compute, directing LLM inference requests across a global fleet while maintaining low latency, high availability, and privacy guarantees.
Role type
Senior IC distributed systems engineer (LLM inference routing)
Builds
Global traffic management, capacity-aware request placement, load balancing, and failover services for Apple Intelligence and Siri AI
Domain
Cloud infrastructure + Large Language Model (LLM) inference serving
Deliverable
production ML models
Required skills
Distributed systems design, LLM inference serving, Network protocols, High-performance server software, Cloud deployment (Kubernetes), System benchmarking
Preferred skills
LLM inference frameworks (llm-d, vLLM, SGLang), KV cache aware routing, Prefill/decode disaggregation, Cryptographic protocols, Secure protocols (TLS, X.509)
Technologies
Java, C/C++, C#, Go, Swift, Kubernetes, TLS, X.509
Responsibilities
Design, build, and operate PCC's inference routing and request scheduling services including global traffic management and failover. Develop adaptive routing strategies that respond to changing demand, fleet health, and available compute capacity. Build admission control, prioritization, and overload management mechanisms. Partner with model serving and infrastructure teams to evaluate routing strategies for prefill/decode disaggregation. Design network protocols and implement high-performance server software for inference request dispatch. Define performance metrics and benchmark routing strategies under realistic workloads. Own projects from architecture through production rollout, partnering with Siri AI, Apple Intelligence, and security teams.
