⚠ INFORMATION
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.

Course: Deep Learning optimized for Jean Zay (DLO-JZ)

---

Manager : Léo Mantegazza
Instructors : Members of the IDRIS support team

The aim of this training is to review the main current techniques for optimising machine learning in Deep Learning, with the goal of scaling on a supercomputer. The associated issues of acceleration and distribution on multiple GPUs will be addressed, from a system and algorithmic point of view.

Next session will be supported by Minerva.

Register for this training

Target audience

---

Target audience

This training is intended for AI users of Jean Zay or people who master the fundamentals of Deep Learning and who wish to learn about the challenges of scaling up.

Prerequisites

---

Prerequisites

The required prerequisites are:

• master the concepts of learning in Deep Learning
• master Python
• have basics in PyTorch for the proper execution of the practical work

Duration and practical info

---

Duration and practical info

Unless otherwise specified, IDRIS courses are taught in French.

This training lasts 4 days:

• from 09:00 to 12:00 and
• from 13:00 to 17:00.

It takes place exclusively in-person at the premises of IDRIS in Orsay (91).

Attendance

Minimum : 10 people ;
Maximum : 16 people.

Course content

---

Course content

This training is dedicated to the multi-GPU scaling of the training of a Deep Learning model. The common thread of the practical aspects of the training is the optimized scaling (acceleration, distribution) of a model training on the ImageNet dataset, in PyTorch. For this, participants will be led to code and submit their calculations on Jean Zay applying the different concepts covered during the course sections.

Plan

Day 1

• Presentation of the DLO-JZ training
• The Jean Zay supercomputer
• The challenges of scaling up
• GPU acceleration
• Mixed precision
• Tensor formats optimization (channels last memory format)
• PyTorch Profiler

Day 2

• Dataloader optimization
• Distributed training: general concepts and data parallelism
• Data augmentation

Day 3

• Storage and format of input data (webdataset)
• Large batch training techniques (lr scheduler, large batch optimizers, ...)
• Compilation (JIT)

Day 4

• Best practices (small models)
• Managing large models and Fully Sharded Data Parallelism (FSDP)
• Model parallelism
• Training visualization tools (TensorBoard, MLflow, Weights & Biases, ...)
• Optimization and hyperparameter search techniques

INFO

For an efficient execution of the practical parts, these will take place on the Jean Zay supercomputer. A workstation with access to the IDRIS supercomputer is provided to the learners. Experience in using a supercomputer, as well as prior access to it, are not required.

Course materials

---

Course materials

INFO

All course materials, including slides, notes, and practical exercises, are provided under the following license: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). For more details on the license, please consult this page.

Slides (standalone and annotated slides)

---

• Introduction, Jean Zay — and annotated version
• Challenges of Scaling — and annotated version
• Code Review — and annotated version
• GPU Computing and Tensor Cores — and annotated version
• Profiler — and annotated version
• DataLoader Optimization — and annotated version
• Distribution and Data Parallelism — and annotated version
• Storage Spaces and Data Format — and annotated version
• Optimizers & Large Batches — and annotated version
• Compiler — and annotated version
• Good Practice and State Of The Art — and annotated version
• Large Models — and annotated version
• Large Model Parallelisms — and annotated version
• Visualization Tools — and annotated version
• Hyperparameters Optimization — and annotated version
• Conclusion — and annotated version

Practical work

---

• Hands-on materials
• Quizz

Upcoming sessions

---

To view the dates of the upcoming sessions for this training, go to the following page:

🗓️ Upcoming sessions

Registration

CNRS/French university staff	External participants
Are you a member of CNRS or a French university staff? Your registration is free via our server.	Our training is aimed at all professionals from companies, public bodies and individuals.
Register as a public institution staff	Register as an external participant