Research Scientist - Pre-training
Core
Build a distributed pretraining system for large models running on heterogeneous consumer-grade devices connected via the internet.
Role type
Senior Research Scientist (Distributed Systems & ML)
Builds
A decentralized training infrastructure for frontier-scale models
Domain
Distributed Machine Learning, Protocol Learning
Deliverable
production ML models
Required skills
distributed training, model parallelism, PyTorch, Python, fault tolerance, performance optimization
Preferred skills
large language model training, P2P networking, RL, inference systems
Technologies
PyTorch, FSDP, DeepSpeed, Megatron
Responsibilities
Implement and optimize data, pipeline, and tensor parallelism for large models; reduce communication overhead in high-latency networks; build robust checkpointing and state synchronization for node churn; develop monitoring tools for throughput and model quality across hundreds of devices.
