Technical Lead Manager - Training Runtime, Data(set) Movement
Core
Design and build a unified dataset read platform and reliability infrastructure for OpenAI's large-scale model training runs (e.g., GPT-5.5), ensuring fast, correct, and reproducible data access across distributed clusters.
Role type
Senior hands-on Technical Lead Manager (Data Movement & Training Runtime)
Builds
Unified dataset read platform, dataset APIs, storage contracts, versioning models, debugging tools, and visualizers for text/multimodal/RL data.
Domain
Machine Learning Systems, Distributed Storage, High-Throughput Data Loading
Deliverable
production ML models | infrastructure
Required skills
Distributed systems architecture, API design, reliability engineering, stateful iteration, checkpoint/restart semantics, high-throughput storage reads, Python, Rust or C++
Preferred skills
Experience with torch.utils.data, multimodal/video/RL data pipelines, lower-level systems code, leading teams without losing hands-on edge
Technologies
Python, Rust, C++, torch.utils.data
Responsibilities
Design and build a unified dataset read platform for multiple training frameworks; Define dataset APIs, storage-format expectations, registration/versioning, and migration paths; Build reliability into the read path including stateful iteration, caching, fast restart, and recovery; Build terminal and web-based visualizers for inspecting pipeline data; Write and review production code in core data loading, service, caching, and reliability paths; Partner with teams on training frameworks, reinforcement learning, multimodal models, storage, runtime, and cluster infrastructure.
Seniority
Senior, hands-on IC with leadership responsibilities
