Led by Huihuo Zheng, Corey Adams, and Zhen Xie from ALCF
This section of the workshop will introduce to you the methods we use to run distributed deep learning training on ALCF resources like Theta and ThetaGPU.
We show distributed training using three frameworks:
- Horovod (for TensorFlow and PyTorch), and
- DistributedDataParallel (DDP) (for PyTorch only).
- DeepSpeed