Senior System Architect, Infrastructure Reliability
Core
Architect and build an automated framework to ingest telemetry from CPU and GPU clusters, identify root causes of job failures in real-time, and distinguish between hardware faults, infrastructure instability, and software defects for EDA workloads.
Role type
Senior System Architect (Infrastructure Reliability)
Builds
Automated failure attribution frameworks and diagnostic pipelines for HPC/EDA clusters
Domain
High-Performance Computing (HPC) / EDA / Accelerated Computing
Deliverable
production ML models | infrastructure
Required skills
Distributed systems programming, C++, Python, CPU architecture (x86/ARM), Cluster resource management (Slurm/Kubernetes), Root cause analysis (RCA), Machine learning for classification, Linux kernel internals, GPU monitoring (DCGM/NVML)
Preferred skills
Low-level diagnostics (/dev/mcelog, dmesg), Checkpoint/restore technologies (CRIU), Non-intrusive monitoring, NUMA optimization
Technologies
C++, Python, Slurm, Kubernetes, Linux, DCGM, NVML, CUDA
Responsibilities
Architect scalable flight recorder systems for high-fidelity state capture; Build automated diagnostics correlating GPU XID errors, PCIe failures, and system events; Implement low-overhead distributed logging and tracing; Develop ML-based heuristics to classify failure types; Engineer resiliency signals for proactive job migration or checkpointing
Seniority
Senior, hands-on IC
