Senior AI Research Engineer - LLM Training & Inference Infrastructure
Core
Build and operate distributed runtime systems for large-scale LLM training, reinforcement learning, and inference to support autonomous agents in cloud observability and security.
Role type
Senior IC distributed systems engineer (ML infrastructure)
Builds
Distributed training/inference infrastructure, simulation environments, RL training loops, and evaluation sandboxes for LLMs
Domain
Cloud observability, security, large-scale distributed systems, GPU clusters
Deliverable
infrastructure
Required skills
distributed systems, Python, systems programming (Rust/C++/Go), GPU cluster operations, scheduling/orchestration, debugging/profiling, incident diagnosis
Preferred skills
Mixture-of-Experts models, parallelism strategies (data/tensor/pipeline/expert), memory optimization, checkpointing, failure recovery, RL rollout systems, collective communication (NCCL/RDMA), GPU benchmarking, open-source ML contributions
Technologies
Ray, Kubernetes, Slurm, Megatron-LM, TorchTitan, DeepSpeed, vLLM, SGLang, NCCL, RDMA
Responsibilities
Design and operate distributed training/inference infrastructure; scale training with parallelism and memory optimization; build inference/rollout systems for RL; maintain execution environments and sandboxes; own continuous performance validation and dashboards; improve researcher experience (scheduling, quotas, GPU utilization); collaborate on cluster capacity and networking; investigate performance/reliability issues across hardware and software
Seniority
Senior, hands-on IC
