Skip to main content
⚠ INFORMATION
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.

Running a multi-GPU MPI CUDA-aware and GPUDirect batch job

Jobs are managed across all nodes by the Slurm software.

To submit a distributed multi-GPU MPI CUDA-aware and GPUDirect batch job on Jean Zay, you need to create a submission script based on the two examples provided below.

Attention

It is necessary to run the code using the same OpenMPI CUDA-aware library that you used to compile your code. Additionally, using the MPI CUDA-aware GPUDirect feature on Jean Zay requires following a specific initialisation order for CUDA or OpenACC and MPI in the code. Please refer to theMPI CUDA-aware and GPUDirect page.

  1. Create a submission script:
multi_gpu_mpi_cuda-aware.slurm
#!/bin/bash
#SBATCH --job-name=multi_gpu_mpi_cuda-aware # nom du job
# Il est possible d'utiliser uniquement des GPU V100 16 Go ou 32 Go
##SBATCH -C v100-16g                 # decommenter pour reserver uniquement des GPU V100 16 Go
##SBATCH -C v100-32g                 # decommenter pour reserver uniquement des GPU V100 32 Go
# Ici, reservation de 2 noeuds avec 4 GPU par noeud et 10 CPU par tache :
#SBATCH --nodes=2                    # nombre de noeuds
#SBATCH --ntasks-per-node=4          # nombre de taches par noeud (= nombre de GPU par noeud ici)
#SBATCH --gres=gpu:4                 # nombre de GPU par noeud (max 4)
# Le nombre de CPU par tache doit etre adapte en fonction de la partition utilisee. Sachant
# qu'ici on reserve un GPU par tache (soit 1/4 des GPU du noeud), l'ideal est de reserver 1/4
# des CPU du noeud par tache :
#SBATCH --cpus-per-task=10           # nombre de CPU par tache (1/4 des CPU du noeud 4-GPU V100)
# /!\ Attention, "multithread" fait reference à l'hyperthreading dans la terminologie Slurm
#SBATCH --hint=nomultithread         # hyperthreading desactive
#SBATCH --time=00:10:00              # temps maximum d'execution demande (HH:MM:SS)
#SBATCH --output=multi_gpu%j.out     # nom du fichier de sortie (%j est remplacé par le numéro du travail)
#SBATCH --error=multi_gpu%j.out      # nom du fichier d'erreur (ici commun avec la sortie)

# Nettoyage des modules charges en interactif et herites par defaut
module purge

# Chargement des modules
module load ...

# Echo des commandes lancees
set -x

# Le code doit etre compile avec les modules compatibles avec la partition choisie
# Execution du code
srun ./executable_multi_gpu_mpi_cuda-aware
  1. Submit this script using the command sbatch :
sbatch multi_gpu_mpi_cuda-aware.slurm

Remarks:​

  • The error CUDA failure: cuCtxGetDevice() returned during execution probably means that you did not follow the CUDA or OpenACC and MPI initialisation order, as indicated on the MPI CUDA-aware and GPUDirect page.
  • We recommend compiling and running your code in the same software environment by loading the same modules.
  • The option --hint=nomultithread ensures the reservation of physical cores (no hyperthreading).
  • The memory allocated for the job is proportional to the number of CPU cores requested. For example, if you request 1/4 of the physical CPU cores of a node, you will have access to 1/4 of its RAM memory. It is important to be consistent with the configuration of the nodes used to avoid overcharging of hours, while benefiting from the memory to which you are entitled. You can consult our documentation on this subject: Memory allocation with Slurm.
  • In these examples, it is assumed that the executable executable_multi_gpu_mpi_cuda-aware is located in the submission directory, i.e. the directory in which you are located when using the command sbatch : the variable $SLURM_SUBMIT_DIR is automatically set by Slurm.
  • The output file of the computation will also be located in the submission directory. It is created at the beginning of the job execution; editing or modifying it during the job execution may disrupt it.
  • The module purge is made necessary by Slurm's default behaviour: the modules you have loaded in your environment at the time you run sbatch are taken into account in the submitted job.
  • The use of the command srun is essential when you request a multi-task execution. We advise against using mpirun on Jean Zay, only srun guarantees a distribution in accordance with the resource specifications requested in your submission file.
  • All jobs have resources defined by a partition and a "Quality of Service" QoS (Quality of Service) set by default in Slurm. You can modify these limits by specifying a partition and/or a QoS as indicated in our documentation detailing GPU partitions and QoS.
  • For multi-project accounts as well as those with CPU and GPU hours, it is essential to specify the hour allocation on which to deduct the computing hours of the job as indicated in our documentation detailing the management of computing hours to ensure that the hours consumed by your jobs are deducted from the correct allocation.

Your opinion matters!

To give your feedback, report an error, or suggest an improvement, click here:

quick anonymous questionnaire

This questionnaire is temporary and will take less than a minute, so take the opportunity!