Reliability Engineer, Supercomputing
Core
Ensure the reliability of a GPU supercomputing fleet by diagnosing hardware issues, managing drivers/firmware/OS interfaces, and resolving root causes to enable large-scale AI research.
Role type
Reliability Engineer (Supercomputing)
Builds
Reliable GPU clusters for frontier AI model training and research
Domain
AI / Supercomputing / Hardware Reliability
Deliverable
production ML models
Required skills
Python or Rust, Linux systems, Kubernetes or Slurm, hardware debugging, vendor engagement, firmware lifecycle management, statistical analysis
Preferred skills
Linux kernel literacy, out-of-band management (BMC/iDRAC/IPMI), GPU hardware health (Xid/NVLink/DCGM), statistical rigor, hardware vendor engagement
Technologies
Python, Rust, Kubernetes, Slurm, Linux, BMC, iDRAC, IPMI, Redfish, DCGM
Responsibilities
Investigate and remediate issues across large GPU clusters; Own drivers, kernel surface, and diagnostics spanning hardware, firmware, and OS; Automate fleet reliability monitoring and error rate analysis; Drive firmware lifecycle including qualification and regression analysis; Engage hardware vendors directly to resolve issues and manage RMA flows; Monitor GPU hardware health signals and implement reliability improvements; Write postmortems and vendor cases
Seniority
Mid-Senior, hands-on IC