Observability & Scale Engineer
Core
Owns observability/monitoring, utilization analysis, demand insight, capacity forecasting, scaling triggers, and recommendations for NVIDIA AI infrastructure and workloads.
Role type
Senior IC observability and capacity planning engineer
Builds
Monitoring dashboards, alerts, thresholds, and operational views for platform teams and leaders
Domain
Cloud infrastructure, AI/ML workloads, GPU clusters
Deliverable
dashboards & analysis
Required skills
NVIDIA DCGM, Prometheus, Grafana, GPU Monitoring, Observability, Capacity Planning, Utilization Analysis, Performance Analysis, Forecasting, Scaling Triggers
Preferred skills
Run:ai Scheduler, Kubernetes scheduling, queue analysis, AI workload demand, Python, APIs, data analytics, dashboard automation
Technologies
DCGM, Prometheus, Grafana, Kubernetes, Linux, NVIDIA GPUs
Responsibilities
Implement and maintain monitoring using DCGM, Prometheus, Grafana, and related telemetry sources; Define service health, GPU, fabric, storage, queue, workload, and utilization measures; Analyze utilization, queue depth, workload demand, bottlenecks, and capacity trends; Develop forecasts, scaling triggers, and evidence-based capacity recommendations; Partner with platform operations, FDE, architecture, finance, and customer teams on demand and growth planning; Validate monitoring coverage during deployment and maintain baselines through the lifecycle; Improve data quality and automate recurring performance and capacity reporting.