Software Engineer, Fleet Hardware Health
Core
Building and maintaining automation systems for provisioning, monitoring, and managing large-scale server fleets to ensure hardware reliability and uptime for AI compute infrastructure.
Role type
Senior IC infrastructure software engineer (fleet hardware health)
Builds
Automated provisioning and management systems for server fleets; monitoring tools for health, performance, and lifecycle events.
Domain
Data center infrastructure / High-performance computing (HPC) / Distributed systems
Deliverable
production ML models | infrastructure
Required skills
Python, Go, Linux, networking, server hardware, SQL, PromQL, Pandas, system-level investigation, automation development
Preferred skills
Low-level hardware component knowledge (PCIe, Infiniband, power management, kernel tuning), hardware management protocols (IPMI, Redfish), HPC or distributed systems experience, hardware design/management experience, monitoring tools (Prometheus, Grafana)
Technologies
Python, Go, Linux, SQL, PromQL, Pandas, PCIe, Infiniband, IPMI, Redfish, Prometheus, Grafana
Responsibilities
Build and maintain automation systems for provisioning and managing server fleets; Develop tools to monitor server health, performance, and lifecycle events; Identify and fix performance bottlenecks and inefficiencies; Continuously improve automation to reduce manual work; Collaborate with clusters, networking, and infrastructure teams; Partner with external operators to ensure quality.
Seniority
Senior, hands-on IC
