SDE2 - API Platform
Core
Building and evolving a multi-tenant LLM API platform handling routing, failover, streaming, metering, and billing for developers and enterprises.
Role type
Senior IC backend engineer (API platform)
Builds
OpenAI-compatible LLM APIs, streaming infrastructure, and commercial metering/billing layers
Domain
AI infrastructure, distributed systems, backend services
Deliverable
production ML models
Required skills
Python (asyncio, FastAPI), Go, low-latency multi-tenant backend services, streaming systems (SSE, WebSockets), PostgreSQL, Redis, Kubernetes
Preferred skills
LLM integration experience, inference server familiarity (vLLM, SGLang, Dynamo), metering/billing infrastructure, early-stage startup experience
Technologies
FastAPI, Python, Go, PostgreSQL, Redis, Kubernetes, vLLM, SGLang, Dynamo
Responsibilities
Building the Python-to-Go rewrite of the API platform, implementing streaming infrastructure for Chat Completions, managing routing and failover across model deployments, engineering the commercial metering and billing layer, ensuring performance and reliability for multi-tenant workloads, maintaining observability and running load tests
Seniority
Senior, hands-on IC