Software Engineer, Model Serving System
Core
Build and optimize the end-to-end path for Microsoft AI's large language models (LLM) to run reliably in production on the Azure cloud platform.
Role type
Senior IC software engineer (LLM serving systems)
Builds
Production serving environments for LLMs on Azure
Domain
Cloud infrastructure + Large Language Model serving
Deliverable
production ML models
Required skills
LLM inference/serving systems (vLLM, SGLang, TensorRT-LLM), distributed serving concepts (TP/PP/DP, KV cache), systems/debugging on GPU clusters, Python, C++
Preferred skills
Azure ML/Foundry, Kubernetes, multi-adapter serving, observability tools (Grafana, Prometheus, Kusto), evaluation/benchmarking harnesses, GPU topology and memory management
Technologies
vLLM, SGLang, TensorRT-LLM, Azure ML, Foundry, Kubernetes, Grafana, Prometheus, Kusto
Responsibilities
Integrate and deploy models to production serving environments with high reliability; Profile and optimize for production SLOs (TTFT, TPOT, throughput, QPS) on modern GPU hardware; Design and run benchmarks using real usage patterns to measure serving performance; Build and improve dashboards and telemetry to track utilization, latency, and SLA compliance
Seniority
Senior, hands-on IC