Senior Software Engineer, AI Data
Core
Design and build the AI data layer for a multi-cloud GPU fleet, managing datasets, checkpoints, and scratch data placement, caching, and tiering across storage backends to support high-throughput training and RL workloads.
Role type
Senior IC infrastructure engineer (AI data platform)
Builds
Kubernetes-native data platform, streaming loaders, distributed caches, and quota/capacity management systems for GPU training
Domain
AI/ML infrastructure, distributed storage, high-performance computing
Deliverable
infrastructure
Required skills
Go, Rust, C/C++, Python, distributed file systems, object storage, I/O performance optimization, Kubernetes, capacity planning
Preferred skills
Large-scale AI training data loaders, checkpointing at scale, distributed cache systems, RL workload storage patterns
Technologies
Kubernetes, Slurm, Lustre, WEKA, VAST, GPFS, Ceph, S3, NVMe
Responsibilities
Design data placement and caching strategies for multi-cloud GPU fleets; Build high-throughput data paths and streaming loaders; Implement Kubernetes-native APIs and storage integrations; Manage quotas, capacity, and data lifecycle across research teams; Build I/O telemetry and access analytics; Ensure reliability through on-call and incident analysis
Seniority
Senior, hands-on IC
