High Throughput Hyperparameter Optimization of ML/DL models on KSL platforms

Overview

Hyperparameter optimization (HPO), a.k.a hyperparameter tuning, is a common and compute-hungry step in the development of a reliable Machine Learning/Deep Learning model. It is an iterative process requiring lots of retraining of the model with different parameters. In some instances, a significant number of independent SLURM jobs may be needed to get to a good enough combination of parameters. Bad trials should also be curtailed in order to get to a good guess of parameters. This all means orchestrating HPO experiments manually and managing them over long periods by the user.

I will introduce a framework to consolidate and automate trials of an HPO experiment with smartness to stop bad-performing trials early and schedule them to steer towards their “good” values in the parameter space. Ray Tune is a scalable Python library for running large-scale distributed HPO experiments. I will demonstrate how to build and run Ray Tune on Shaheen2, Ibex CPUs, and Ibex GPUs for Machine Learning and Deep Learning experiments. A catalog of available search and scheduling algorithms in Tune will also be presented. We will shortly discuss visualizing the training progress and integration of Ray Tune with the Weights & Biases library to keep track of experiments.

This is a hands-on session, so please bring your laptops with a Linux terminal installed

Learning Outcomes

After attending the training, you will be able to:

• Understand the considerations when refactoring training scripts to scale from 1 to N GPUs
• Understand the data management related to distributed training jobs
• Familiarize how to launch distributed training jobs on Ibex resources
• Understanding the scaling characteristics of your distributed training workload

A Quiz will be conducted after the training, which is mandatory to submit to ensure the continued use of KSL resources.