Software Engineer, Model Routing & Inference
Core
Building the inference platform that powers every AI interaction in the product, making it faster, more reliable, and cost-effective at massive scale.
Role type
Senior IC software engineer (distributed systems & inference serving)
Builds
Inference gateway, cross-provider failover systems, routing backpressure and admission control mechanisms
Domain
AI infrastructure, distributed systems, cloud-scale inference serving
Deliverable
production ML models
Required skills
high-throughput distributed systems design, low-latency inference serving, traffic routing, real-time data pipelines, GPU utilization optimization, capacity planning, production system architecture
Preferred skills
cost/performance tradeoff analysis at scale, handling millions of requests, decision-making in ambiguous technical scenarios
Technologies
inference serving frameworks, API abstraction layers, load balancing systems
Responsibilities
Design and evolve the inference gateway to abstract provider API semantics, implement intelligent cross-provider failover logic, design routing backpressure and admission control to prevent traffic cascades
Seniority
Senior, hands-on IC
