CareerPlan

Senior AI Infrastructure Engineer, Observability

Shenton Way, SingaporeFull-time2026-11-12 → 2026-11-19

Core

Establish health standards, diagnostics, and operational guidance for GPU infrastructure to ensure safe commissioning, early failure detection, and rapid recovery for customer and internal AI workloads.

Role type

Senior AI Infrastructure Engineer (Observability)

Builds

GPU health dashboards, alerts, diagnostic checks, runbooks, and operational knowledge bases

Domain

AI Infrastructure / GPU Cloud / HPC

Deliverable

production ML models | infrastructure

Required skills

GPU fault diagnosis, Linux server fundamentals, Python automation, infrastructure telemetry analysis, PromQL/LogQL, Grafana, DCGM, NCCL testing, thermal/power management

Preferred skills

Large GPU systems with tight interconnects, multi-site operations, AI cloud/HPC background, ML failure prediction, AI assistant integration

Technologies

PromQL, LogQL, Grafana, DCGM, NCCL, Python, BMC/management interfaces

Responsibilities

Define GPU and host health criteria and service-readiness gates; Publish golden dashboards, alerts, and health checks; Create validation jobs for performance confirmation; Isolate faults between GPU, cooling, host, network, or power; Monitor silent failures like ECC errors and NVLink retries; Partner with operations and commissioning teams on incident response and knowledge sharing

Seniority

Senior, hands-on IC