ML Engineer (Training Infra), Foundational Models
Core
Build and optimize distributed training infrastructure for large-scale foundational models, ensuring high throughput, stability, and reliability across GPU clusters.
Role type
Senior ML Infrastructure Engineer (Training Systems)
Builds
Distributed training stacks for frontier AI models
Domain
AI Infrastructure / High-Performance Computing
Deliverable
infrastructure
Required skills
Distributed training frameworks (Megatron-LM, DeepSpeed, FSDP, NeMo), GPU architecture & CUDA programming, PyTorch internals, Custom kernel development (CUDA, Triton), Cluster orchestration (Slurm, Kubernetes), Profiling tools (Nsight, PyTorch profiler)
Preferred skills
Training models at 10B+ parameters, Mixed precision training (BF16, FP8), Quantization-aware training, Open-source contributions to training ecosystem, First-author papers on training systems
Technologies
Megatron-LM, DeepSpeed, FSDP, NeMo, CUDA, Triton, PyTorch, Nsight, Slurm, Kubernetes, NCCL, vLLM
Responsibilities
Design and implement parallelism strategies (data, tensor, pipeline, sequence, expert); Profile and optimize end-to-end training throughput; Write and tune custom GPU kernels; Own reliability of long-running training jobs (fault tolerance, checkpoint integrity); Partner with researchers and data teams to optimize training efficiency
Seniority
Senior, hands-on IC