Accelerate HuggingFace

⚠ INFORMATION
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.

Accelerate is a HuggingFace library that facilitates the use of accelerators (GPUs in the case of Jean Zay).

Single-node

You can normally already use Accelerate on a single node easily with HuggingFace's documentation.

Multi-node

For multi-node usage, it is necessary to generate one or more YAML configuration files and pass them to the Accelerate code with the --config_file parameter (or --config for older versions). Additionally, one execution per node is required, with the correct configuration file for each execution.

You can easily use Accelerate in a multi-node setup on Jean Zay thanks to the idr_accelerate launcher. This tool, implemented by IDRIS, is available on most Jean Zay modules.

This launcher handles the previously mentioned constraints for you by creating and executing the Accelerate codes. It replaces the Accelerate launcher as follows: accelerate launch train.py becomes idr_accelerate train.py.

To use idr_accelerate in a multi-node setup, you simply need to configure 1 task per node and use srun to launch one execution per node. Here is a simple example:

#!/bin/bash
#SBATCH --job-name=accelerate_multi-node
#SBATCH --gres=gpu:4
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=2
#SBATCH --cpus-per-task=40
#SBATCH --hint=nomultithread
#SBATCH --time=02:00:00
#SBATCH --qos=qos_gpu-dev
#SBATCH --account=account@v100

## load module
module purge
module load pytorch-gpu

srun idr_accelerate train.py --lr 0.5 --epochs 10

A complete example of using Accelerate for DDP is available on our LLM examples git.

Deepspeed via Accelerate

Accelerate allows for a "simplified" use of DeepSpeed. This enables you to take advantage of certain optimisations like ZeRO. The deepspeed accelerate documentation explains the basic operation and provides some examples.

You can use DeepSpeed in 3 different ways with idr_accelerate:

# On donne uniquement des flags d'utilisation (simple mais peu d'options possibles)
srun idr_accelerate --use_deepspeed --zero_stage 2 --mixed_precision fp16 train.py --lr 0.5

# On donne un fichier de configuration accelerate contenant les paramètres DeepSpeed à utiliser
srun idr_accelerate --config_file configs/accelerate_deepspeed.yaml train.py --lr 0.5

# On donne un fichier de configuration accelerate qui pointe vers un fichier de configuration DeepSpeed
# (méthode la plus complexe mais aussi avec le plus d'options)
srun idr_accelerate --config_file=configs/accelerate_deepspeed-config.yaml train.py --lr 0.5