Site Reliability Engineer
Core
Design campus monitoring architecture, lead cross-discipline incident response, and build reliability guardrails for a large AI training campus spanning compute, network, storage, power, and cooling.
Role type
Senior Site Reliability Engineer (Campus Reliability)
Builds
Monitoring architecture, incident response playbooks, and reliability guardrails for GPU clusters and facility systems
Domain
AI Infrastructure / High-Performance Computing / Data Center Operations
Deliverable
production ML models
Required skills
Fleet-scale observability design, incident command leadership, change management for firmware/BIOS, cross-functional facilitation, SLO/SLI definition, scripting (Python/Bash), systems programming (C/C++/Go/Rust)
Preferred skills
Power plant or nuclear plant operations experience, direct liquid cooling expertise, AI/ML infrastructure experience, fleet firmware management, reliability engineering methods (MTBF/AFR)
Technologies
Prometheus, Grafana, DCGM, SCADA, BMS, EPMS, DCIM, InfiniBand, RoCE, Redfish, IPMI, Kubernetes, Slurm
Responsibilities
Own monitoring architecture and signal quality across GPU and facility telemetry; Provide technical incident leadership and bridge coordination; Run blameless postmortems and drive corrective actions; Gate high-risk changes including fleet-scale firmware rollouts; Lead cross-functional reliability projects spanning compute, network, and facility boundaries; Partner with Hardware Failure Analysis on systemic failure classes; Define error budgets and availability objectives; Participate in on-call rotations for SEV-class events
Seniority
Senior, hands-on IC
