Senior AI Infrastructure Engineer, Observability
Core
Establish health standards, diagnostics, and operational guidance for GPU infrastructure to ensure safe commissioning, early failure detection, and rapid recovery for customer and internal AI workloads.
Role type
Senior AI Infrastructure Engineer (Observability)
Builds
GPU health dashboards, alerts, diagnostic checks, runbooks, and operational knowledge bases
Domain
AI Infrastructure / GPU Cloud / HPC
Deliverable
production ML models | infrastructure
Required skills
GPU fault diagnosis, Linux server fundamentals, Python automation, infrastructure telemetry analysis, PromQL/LogQL, Grafana, DCGM, NCCL testing, thermal/power management
Preferred skills
Large GPU systems with tight interconnects, multi-site operations, AI cloud/HPC background, ML failure prediction, AI assistant integration
Technologies
PromQL, LogQL, Grafana, DCGM, NCCL, Python, BMC/management interfaces
Responsibilities
Define GPU and host health criteria and service-readiness gates; Publish golden dashboards, alerts, and health checks; Create validation jobs for performance confirmation; Isolate faults between GPU, cooling, host, network, or power; Monitor silent failures like ECC errors and NVLink retries; Partner with operations and commissioning teams on incident response and knowledge sharing
Seniority
Senior, hands-on IC