Distributed Deep Learning on KSL Platforms

Overview

With the increasing complexity and size of both Deep Learning (DL) models and datasets, the computational cost of training these model can be non-trivial, ranging from a few tens of hours to even several days. Exploiting parallelism exhibited inherently in the training process of DL models; we can distribute training on multiple GPUs on single or multiple nodes of Ibex. We will survey the available distributed training frameworks (e.g. PyTorch DDP, PyTorch Lightning, Jax) along with demonstrations and hands-on exercises on how to run them on Ibex resources.

Learning Outcomes

After attending the training, you will be able to:

• Understand the considerations when refactoring training scripts to scale from 1 to N GPUs
• Understand the data management related to distributed training jobs
• Familiarize how to launch distributed training jobs on Ibex resources
• Understanding the scaling characteristics of your distributed training workload

A Quiz will be conducted after the training, which is mandatory to submit to ensure the continued use of KSL resources.