Member of Technical Staff, Distributed Training Performance
Core
Improving performance of Megatron-based pretraining systems across thousands of GPUs to enable seamless, real-time multimodal intelligence.
Role type
Senior IC distributed training performance engineer (multimodal AI)
Builds
High-throughput pretraining runs for multimodal models (text, audio, vision)
Domain
AI/ML infrastructure, large-scale distributed training
Deliverable
production ML models
Required skills
Distributed training systems internals, Megatron-LM/NeMo frameworks, GPU kernel optimization (CUDA/Triton/CUTLASS), parallelism strategies (data/tensor/pipeline/expert), GPU profiling, mixed-precision training (BF16/FP8), Python/C++ engineering
Preferred skills
Mixture-of-experts training at scale, GPU communication libraries (NCCL), fault-tolerant training, cluster schedulers (Slurm/Kubernetes)
Technologies
Megatron-LM, NeMo, CUDA, Triton, CUTLASS, NCCL, Slurm, Kubernetes
Responsibilities
Own MFU and step time for pretraining runs, write/integrate custom kernels for attention and MoE routing, overlap communication with compute, implement model sharding and parallelism strategies, profile GPU kernels and communication, develop reusable training components, investigate performance regressions, improve checkpointing and observability
Seniority
Senior, hands-on IC