Principal High-Performance LLM Training Engineer
Core
Drive performance of large-scale AI training and post-training workloads across NVIDIA's full hardware and software stack, optimizing frontier-scale LLM workloads running on thousands of GPUs.
Role type
Principal High-Performance LLM Training Engineer
Builds
Production-quality software, tools, models, benchmarks, and analysis infrastructure for NVIDIA's AI software stack.
Domain
AI/ML Systems, High-Performance Computing, GPU Architecture
Deliverable
production ML models
Required skills
Large-scale AI training systems, GPU performance optimization, distributed systems, high-performance computing, ML frameworks, compilers/runtimes, hardware/software co-design, transformer-based model optimization, reinforcement learning, fine-tuning, profiling, tracing, benchmarking, performance modeling
Preferred skills
Technical leadership, cross-functional influence, mentorship
Technologies
PyTorch, JAX, NeMo, NeMo RL, CUDA, GPU architecture, distributed training (data/tensor/pipeline/expert/sequence parallelism, activation checkpointing, mixed precision)
Responsibilities
Lead end-to-end performance analysis and optimization of LLM pre-training and post-training workloads; Identify and remove bottlenecks across compute, memory, communication, scheduling, parallelism, and kernel efficiency; Develop tools and infrastructure to improve training performance and developer velocity; Build performance models and simulation methodologies to guide architecture decisions; Serve as technical authority for AI training performance; Translate workload insights into hardware and software recommendations; Mentor engineers and establish best practices for large-scale AI performance.
Seniority
Principal, hands-on IC with technical leadership
