CareerPlan

Member of Technical Staff, Distributed Training Performance

HQFull-time$180k–$450k2026-11-10 → 2026-11-19

Core

Improving performance of Megatron-based pretraining systems across thousands of GPUs to enable seamless, real-time multimodal intelligence.

Role type

Senior IC distributed training performance engineer (multimodal AI)

Builds

High-throughput pretraining runs for multimodal models (text, audio, vision)

Domain

AI/ML infrastructure, large-scale distributed training

Deliverable

production ML models

Required skills

Distributed training systems internals, Megatron-LM/NeMo frameworks, GPU kernel optimization (CUDA/Triton/CUTLASS), parallelism strategies (data/tensor/pipeline/expert), GPU profiling, mixed-precision training (BF16/FP8), Python/C++ engineering

Preferred skills

Mixture-of-experts training at scale, GPU communication libraries (NCCL), fault-tolerant training, cluster schedulers (Slurm/Kubernetes)

Technologies

Megatron-LM, NeMo, CUDA, Triton, CUTLASS, NCCL, Slurm, Kubernetes

Responsibilities

Own MFU and step time for pretraining runs, write/integrate custom kernels for attention and MoE routing, overlap communication with compute, implement model sharding and parallelism strategies, profile GPU kernels and communication, develop reusable training components, investigate performance regressions, improve checkpointing and observability

Seniority

Senior, hands-on IC