Jean Zay: GPU Slurm partitions

All the DARI or Dynamic Access projects with GPU hours have Slurm partitions available to them on Jean Zay.

The projects with V100 GPU hours have access by default to a partition which permits using all types of four-GPU accelerated nodes with 160 GB of memory. By default, the execution time is 10 minutes and cannot exceed 100 hours (--time=HH:MM:SS ≤ 100:00:00: see below V100 QoS).

This partition includes both the Nvidia V100 GPUs having 16 GB of memory and the Nvidia V100 GPUs having 32 GB of memory; If you wish to be limited to only one type of GPU, you must specify this by adding one of the following SLURM directives to your scripts:

• #SBATCH -C v100-16g # to select nodes having GPUs with 16 GB of memory (i.e. gpu_p3)
• #SBATCH -C v100-32g # to select nodes having GPUs with 32 GB of memory (i.e. gpu_p1)

Important note: If your job can run on either type of the above nodes, we recommend that you do not specify a type of node (neither -C v100-16g nor -C v100-32g) in order to limit the waiting time of your jobs in the queue.

The gpu_p2 partition is accessible to all researchers who have obtained V100 GPU hours via Dynamic Access (AD) or Regular Access (DARI projects). This partition allows launching jobs on the Jean Zay eight-GPU accelerated nodes. These nodes are equipped with Nvidia V100 GPUs with 32 GB of memory. The execution time by default is 10 minutes and it cannot exceed 100 hours (--time=HH:MM:SS ≤ 100:00:00; see below V100 QoS).
This partition has nodes with 360 GB of RAM and others with 720 GB. Depending on the amount of memory required by your code, you can target one of the following two subpartitions:

• The gpu_p2s subpartition gives access to the V100 eight-GPU nodes with 360 GB of memory.
• The gpu_p2l subpartition gives access to the V100 eight-GPU nodes with 720 GB of memory.

Important note: If your job does not need more than 360GB of memory, it is best not to specifically target a given node type by specifying the –partition=gpu_p2 option in order to limit the queue time of your jobs.

The gpu_p5 partition is only accessible to the researchers who have obtained A100 GPU hours via Dynamic Access (AD) or Regular Access (DARI projects). It allows calculations to be launched on the 52 Jean Zay octo-GPU accelerated nodes which are equipped with Nvidia A100 GPUs interconnected by a SXM4 socket and having 80 GB of memory per GPU. By default, the execution time is 10 minutes and cannot exceed 20 hours (--time=HH:MM:SS ≤ 20:00:00, see below A100 QoS). To use this partition, you must specify the SLURM directive #SBATCH -C a100 in your scripts.

Warning: These nodes include EPYC 7543 Milan AMD processors (64 cores per node), unlike other nodes which feature Intel processors. You must therefore load arch/a100 module (module load arch/a100) first in order to have access to modules compatible with this partition and to recompile your codes.

The gpu_p6 partition is accessible to the researchers who have obtained H100 GPU hours via Dynamic Access (AD) or Regular Access (DARI projects). It allows calculations to be launched on the 364 Jean Zay quadri-GPU accelerated nodes which are equipped with Nvidia H100 GPUs interconnected by a SXM5 socket and having 80 GB of memory per GPU. By default, the execution time is 10 minutes and cannot exceed 100 hours (--time=HH:MM:SS ≤ 100:00:00. see below H100 QoS). To use this partition, you must specify the SLURM directive #SBATCH -C h100 in your scripts.

Warning: These nodes having a specific hardware architecture, it is necessary to load arch/h100 module (module load arch/h100) first in order to have access to modules compatible with this partition and to recompile your codes.

Other partitions are available, to allow you to perform compilations, file transfers and data pre/post-processing:

• The prepost partition allows launching a job on one of the Jean Zay pre-/post-processing nodes, jean-zay-pp: These calculations are not deducted from your allocation. The execution time by default is 2 hours and cannot exceed 20 hours (--time=HH:MM:SS ≤ 20:00:0, see below).
• The visu partition allows launching a job on one of the Jean Zay visualization nodes, jean-zay-visu: These calculations are not deducted from your allocation. The execution time by default is 10 minutes and cannot exceed 4 hours (–time=HH:MM:SS ≤ 4:00:00, see below).
• The archive partition is dedicated to data management (copying or moving files, creating archive files): The computing hours are not deducted from your allocation. The execution time by default is 2 hours and cannot exceed 20 hours (--time=HH:MM:SS ≤ 20:00:00, see below).
• The compil partition is dedicated to library and binary compilations which cannot be done on the front end because they require too much CPU time: The computing hours are not deducted from your allocation. The execution time by default is 2 hours and cannot exceed 20 hours (--time=HH:MM:SS ≤ 20:00:00, see below).

Important: Be careful about the partition default time limits which are intentionally low. For a long execution, you should specify a time limit for the execution which must stay inferior to the maximum time authorised for the partition and the Quality of Service (QoS) used. To specify the time limits you must use either:

• The Slurm directive #SBATCH --time=HH:MM:SS in your job, or
• The option --time=HH:MM:SS of the commands sbatch, salloc or srun.

The default GPU partition does not need to be specified to be used by all jobs requesting GPUs. All the other partitions, however, must be explicitly specified to be used. For example, to specify the prepost partition, you can use either:

• The Slurm directive #SBATCH --partition=prepost in your job, or
• The option --partition=prepost of the commands sbatch, salloc or srun commands.

Warning: Any job requiring more than one node runs in exclusive mode. The nodes are not then shared but this implies that the hours invoiced are calculated on the basis of the totality of the requistioned nodes, including those which were only partially exploited.
For example, the reservation of 41 CPU cores (or 1 node + 1 core) on the cpu_p1 partition results in the invoicing of 80 CPU cores (or 2 nodes). In the same way, reserving 5 GPUs (or 1 four-GPU node + 1 GPU) on the default GPU partition results in the invoicing of 8 GPUs (or 2 four-GPU nodes). However, the total memory of the reserved nodes is available in both cases (approximately 160 usable GBs per node).

For each job submitted on a compute partition (other than archive, compil, prepost and visu), you may specify a Quality of Service (QoS). The QoS determines the time/node limits and priority of your job.

To specify a QoS which is different from the default one, you can either:

• Use the Slurm directive #SBATCH --qos=<chosen_qos> (for example) in your job, or
• Specify the --qos=<chosen_qos> option of the commands sbatch, salloc or srun.

by replacing <chosen_qos> with the name of the desired QoS.

Note that Qos names differ depending on the partitions used.

• The default QoS for all the GPU jobs: qos_gpu-t3
  • Maximum duration: 20h00 of elapsed time
  • 512 GPU maximum per job
  • 512 GPU maximum per user (all projects combined)
  • 512 GPU maximum per project (all users combined)

• A QoS for longer executions, only available on V100 partitions, and which must be specified to be used (see above): qos_gpu-t4
  • Maximum duration: 100h00 of elapsed time
  • 16 GPU maximum per job
  • 96 GPU maximum per user (all projects combined)
  • 96 GPU maximum per project (all users combined)
  • 256 GPU maximum for the totality of jobs requesting this QoS.

• a QoS reserved only for short executions carried out within the frameworks of code development or execution tests and which must be specified to be used (see above): qos_gpu-dev
  • A maximum of 10 jobs simultaneously (running or pending) per user
  • Maximum duration: 2h00 of elapsed time
  • 32 GPU maximum per job
  • 32 GPU maximum per user (all projects combined)
  • 32 GPU maximum per project (all users combined)
  • 512 GPU maximum for the totality of jobs requesting this QoS.

• The default QoS for all the GPU jobs: qos_gpu_a100-t3
  • Maximum duration: 20h00 of elapsed time
  • 128 GPU maximum per job
  • 256 GPU maximum per user (all projects combined)
  • 256 GPU maximum per project (all users combined)

• a QoS reserved only for short executions carried out within the frameworks of code development or execution tests and which must be specified to be used (see above): qos_gpu_a100-dev
  • A maximum of 10 jobs simultaneously (running or pending) per user
  • Maximum duration: 2h00 of elapsed time
  • 32 GPU maximum per job
  • 32 GPU maximum per user (all projects combined)
  • 32 GPU maximum per project (all users combined)
  • 128 GPU maximum for the totality of jobs requesting this QoS.

• Please note : the A100 partition does not have a QoS allowing the execution of jobs lasting more than 20 hours due to the limited number of A100 nodes available.

• The default QoS for all the GPU jobs: qos_gpu_h100-t3
  • Maximum duration: 20h00 of elapsed time
  • 512 GPU maximum per job
  • 512 GPU maximum per user (all projects combined)
  • 512 GPU maximum per project (all users combined)

• A QoS for longer executions, only available on V100 partitions, and which must be specified to be used (see above): qos_gpu_h100-t4
  • Maximum duration: 100h00 of elapsed time
  • 16 GPU maximum per job
  • 64 GPU maximum per user (all projects combined)
  • 64 GPU maximum per project (all users combined)
  • 192 GPU maximum for the totality of jobs requesting this QoS.

• a QoS reserved only for short executions carried out within the frameworks of code development or execution tests and which must be specified to be used (see above): qos_gpu_h100-dev
  • A maximum of 10 jobs simultaneously (running or pending) per user
  • Maximum duration: 2h00 of elapsed time
  • 32 GPU maximum per job
  • 32 GPU maximum per user (all projects combined)
  • 32 GPU maximum per project (all users combined)
  • 384 GPU maximum for the totality of jobs requesting this QoS.