Senior Software Engineer, AI Data
Core
Design and build the AI data layer for a multi-cloud GPU fleet, managing datasets, checkpoints, and scratch data placement, caching, and tiering across storage backends to support high-throughput training and RL workloads.
Role type
Senior IC infrastructure engineer (AI data platform)
Builds
Kubernetes-native data platform, streaming loaders, distributed caches, and checkpoint management systems for GPU clusters
Domain
High-performance computing, AI/ML infrastructure, multi-cloud storage
Deliverable
production ML models
Required skills
Distributed file systems, object storage at scale, I/O performance optimization, Go/Rust/C/C++, Python, Kubernetes, capacity planning, reliability engineering
Preferred skills
Large-scale AI training data loaders, checkpointing at scale, distributed cache systems, quota/accounting systems, RL workload storage patterns
Technologies
Kubernetes, Slurm, Lustre, WEKA, VAST, GPFS, Ceph, S3, NVMe
Responsibilities
Design data placement and caching strategies for multi-cloud GPU fleets; Build high-throughput data paths and streaming loaders; Implement Kubernetes-native APIs and storage integrations; Manage quotas, capacity, and data lifecycle across research teams; Build I/O telemetry and access analytics; Ensure reliability through on-call support and incident analysis
Seniority
Senior, hands-on IC
