Site Reliability Engineer, Post Training
Core
Build and maintain the reliability, performance, and uptime of large-scale post-training and reinforcement learning (RL) training jobs, clusters, and pipelines.
Role type
Senior Site Reliability Engineer (Post-Training & RL)
Builds
Production-grade training infrastructure for frontier AI models
Domain
AI/ML Infrastructure, Distributed Systems, High-Performance Computing
Deliverable
production ML models
Required skills
Python, Go/C++, Linux internals, Networking fundamentals, Distributed systems debugging, On-call management
Preferred skills
GPU/TPU cluster operations, Post-training/RL techniques (RLHF, PPO, DPO), Distributed training frameworks (PyTorch, Ray), Job schedulers (Slurm, Kubernetes), High-performance networking (InfiniBand, RDMA, NCCL), ML observability tooling
Technologies
PyTorch, Ray, Slurm, Kubernetes, InfiniBand, RDMA, NCCL
Responsibilities
Own reliability and performance of large-scale post-training/RL jobs; Debug failures across accelerators, networking, storage, and schedulers; Build monitoring, alerting, and automated recovery; Improve checkpointing and fault tolerance; Build internal tools to reduce toil; Participate in on-call rotation; Write postmortems.
Seniority
Senior, hands-on IC