CareerPlan

Software Engineer, Model Serving System

Mountain ViewFull-time$120k–$235k2026-11-17 → 2026-11-19

Core

Build and optimize the end-to-end path for Microsoft AI's large language models (LLM) to run reliably in production on the Azure cloud platform.

Role type

Senior IC software engineer (LLM serving systems)

Builds

Production serving environments for LLMs on Azure

Domain

Cloud infrastructure + Large Language Model serving

Deliverable

production ML models

Required skills

LLM inference/serving systems (vLLM, SGLang, TensorRT-LLM), distributed serving concepts (TP/PP/DP, KV cache), systems/debugging on GPU clusters, Python, C++

Preferred skills

Azure ML/Foundry, Kubernetes, multi-adapter serving, observability tools (Grafana, Prometheus, Kusto), evaluation/benchmarking harnesses, GPU topology and memory management

Technologies

vLLM, SGLang, TensorRT-LLM, Azure ML, Foundry, Kubernetes, Grafana, Prometheus, Kusto

Responsibilities

Integrate and deploy models to production serving environments with high reliability; Profile and optimize for production SLOs (TTFT, TPOT, throughput, QPS) on modern GPU hardware; Design and run benchmarks using real usage patterns to measure serving performance; Build and improve dashboards and telemetry to track utilization, latency, and SLA compliance

Seniority

Senior, hands-on IC