Engineering Manager, Private Cloud Compute Inference Routing
Core
Lead the design and operation of global inference routing systems for Apple's Private Cloud Compute (PCC), ensuring low latency and high availability for Siri AI and Apple Intelligence while maintaining privacy guarantees.
Role type
Senior Engineering Manager, distributed systems & AI infrastructure
Builds
Global multi-region inference routing and scheduling services for LLMs
Domain
Cloud infrastructure, AI/ML inference serving, distributed systems
Deliverable
production ML models
Required skills
Large-scale distributed systems leadership, API gateway design, LLM inference serving, load balancing, incident management, architectural strategy, team scaling, capacity planning, observability, container orchestration
Preferred skills
LLM inference platform development (llm-d, vLLM, SGLang), inference AI gateway behaviors, privacy-preserving system design, SLO/SLI frameworks, cross-region capacity routing
Technologies
Java, C/C++, Go, Swift, Kubernetes, TLS, X.509, Oblivious HTTP
Responsibilities
Direct recruitment and development of engineers and managers for PCC inference routing; Establish technical strategy for global traffic management and capacity-aware placement; Oversee delivery of adaptive routing, admission control, and overload management; Evolve inference AI gateway capabilities including authentication and rate limiting; Collaborate on prefill/decode disaggregation and cache locality optimizations; Develop robust operations through observability and SRE practices; Enhance efficiency via benchmarking and capacity-aware scheduling; Provide senior leadership with updates on team health and technical risk; Partner with Siri AI, security, and SRE teams to deliver integrated routing capabilities.
