AI Infra / Engineer - Large-scale Training
Core
Build distributed training systems and execution pipelines for UMA's humanoid robot foundation models across pretraining, mid-training, and post-training phases.
Role type
Senior IC AI Infrastructure Engineer (Large-scale Training)
Builds
Scalable training stacks, distributed execution pipelines, and fault-tolerant systems for robotic foundation models.
Domain
Robotics + Distributed Machine Learning Infrastructure
Deliverable
production ML models
Required skills
Distributed training (DDP, FSDP, Megatron), Python, PyTorch, GPU memory management, High-performance computing (HPC), Numerical debugging, System profiling, Fault tolerance, Data movement optimization
Preferred skills
C++, CUDA, Triton, Custom kernels, Compiler optimization, RL infrastructure, Kubernetes, InfiniBand/RoCE
Technologies
PyTorch, DDP, FSDP, Megatron, FlashAttention, Nsight Systems, Slurm, Kubernetes, InfiniBand, RoCE
Responsibilities
Build distributed pretraining systems across multiple GPUs and nodes; Develop RL and post-training execution infrastructure; Maximize MFU and eliminate bottlenecks in the training loop; Integrate and benchmark accelerated kernels and execution techniques; Build efficient training-time data movement and buffering; Turn research prototypes into maintainable training systems; Diagnose distributed and numerical failures; Build reliable checkpointing and fault-tolerant resumption; Strengthen observability and reproducibility.
Seniority
Senior, hands-on IC
