Senior Slurm Cluster & HPC Engineer
Core
Design, deploy, and operate production Slurm clusters for high-performance GPU computing, managing multi-tenant scheduling, cluster health, and elastic capacity between batch training and Kubernetes inference.
Role type
Senior IC HPC/Cluster Systems Engineer (Slurm & Kubernetes)
Builds
Production Slurm clusters on bare metal and VMs, GPU fabrics (InfiniBand/RoCE/NVLink), and elastic compute capacity for AI training and inference.
Domain
High-Performance Computing (HPC), AI Cloud, Bitcoin Mining Infrastructure
Deliverable
production ML models
Required skills
Slurm cluster architecture and lifecycle management, GPU fabric topology modeling (InfiniBand/RoCE/NVLink), multi-tenant scheduling policies, Kubernetes operator patterns (Slinky), cluster health and reliability engineering, infrastructure as code (Terraform/Ansible), parallel/shared storage systems (Lustre/GPFS/WEKA), Python and Bash automation.
Preferred skills
Go programming, experience with slurm-bridge or CoreWeave SUNK, NVIDIA MIG and DCGM diagnostics.
Technologies
Slurm (slurmctld, slurmdbd, slurmrestd), Kubernetes, Slinky, Terraform, Ansible, Prometheus, Grafana, InfiniBand, RoCE, NVLink, CUDA, Pyxis/Enroot, containerd, MPI/PMIx.
Responsibilities
Design and operate production Slurm clusters with high availability and rolling upgrades; model physical GPU fabric topology for rail-optimized and NVLink domains; enforce multi-tenant security policies and fail-closed defaults; implement Slinky slurm-operator and pilot slurm-bridge for co-scheduling; manage elastic capacity shifting between batch training and Kubernetes inference; build cluster health-check systems for GPU diagnostics and automatic job requeue; deliver reproducible clusters via infrastructure as code; instrument observability and reconcile billing metrics.
Seniority
Senior, hands-on IC
