Software Engineer, ML Training Performance
Core
Design, implement, and operate distributed training, mixed-precision, gradient-checkpointing, and GPU-kernel optimizations for VLM, VLA, and foundation-model training.
Role type
Senior IC machine-learning training performance engineer
Builds
Large-scale training infrastructure for VLM, VLA, and foundation models
Domain
AI/ML, High-Performance Computing, Distributed Systems
Deliverable
production ML models
Required skills
Distributed training optimization, Mixed-precision training, Gradient checkpointing, GPU kernel optimization, Reinforcement learning frameworks, NCCL communication optimization, Profiling tools (Nsight, PyTorch Profiler, XLA/JAX), Python, C++, ML compilers, Hardware/software co-design
Preferred skills
Experience with Megatron-LM, TorchTitan, Verl, MILES, GRPO, PPO algorithms
Technologies
Megatron-LM, TorchTitan, NCCL, NVIDIA Nsight, PyTorch Profiler, XLA, JAX, PyTorch, C++, Python
Responsibilities
Profile and debug training-throughput bottlenecks across compute, compiler, framework, and cluster-networking layers. Implement custom kernels and build benchmarks to validate training-performance optimizations. Collaborate with ML researchers, data engineers, and hardware engineers on requirements and architecture for large-scale training infrastructure.
Seniority
Senior, hands-on IC