Jean Zay: Usage of containers via Singularity

Singularity is a multi-platform open source container system developed by Sylabs. One of the principal advantages of Singularity is the reproducibility of scientific computations. This system is effective and well adapted to both high performance and artificial intelligence computations. Reproducibility requires the use of containers in order to port applications from one system to another. By using Singularity containers, users can work in reproducible environments of their choice and design. These complete environments can be easily copied and executed on the Jean Zay supercomputer.

Singularity is accessible via the module command:

$ module load singularity

To obtain information about the Singularity module, you simply need to use the following command:

$ module show singularity

The directory which will hold the temporary filesystem of the container during its execution will be created under a location set by $SINGULARITY_CACHEDIR. After the module is loaded, the location of the Singularity cache associated to the $SINGULARITY_CACHEDIR environment variable is automatically set in the $SCRATCH disk space. It is strongly recommended, therefore, to not modify the $SINGULARITY_CACHEDIR environment variable in order to prevent exceeding the quotas of the other disk spaces and to simplify the automatic purging.

The work directory from which the Singularity commands (build, pull, run, exec, …) are launched must be the $WORK or $SCRATCH space because it is here that the files or subdirectories containing a large number of voluminous files will be stored.

The containers authorized for execution must be placed in the disk space where the path is set by the environment variable $SINGULARITY_ALLOWED_DIR. This variable defines an unique path per user and this path can't be changed. Copying and deleting containers is done through the command idrcontmgr. This command authorizes or not the copying of the containers on this space, by checking beforehand whether the container obeys security constraints defined by IDRIS. This space is restricted to 20 containers

A container can therefore be copied into the authorized execution space via the command idrcontmgr as follows :

$ idrcontmgr cp my-container.sif

The removal is done in a fairly similar way :

$ idrcontmgr rm my-container.sif

This command also displays the content of the authorized execution space :

$ idrcontmgr ls

Building of Singularity images which require root privileges are not available on Jean Zay. Therefore, it is not possible to build a Singularity image from a definition file. Consequently, for these types of images, the user should create them on another machine and then copy them to Jean Zay.

It is possible, however, to create a Singularity image from one of the Jean Zay front-ends by using existing repositories (Singularity, Docker, …) under an SIF file extension format.

From the $WORK or $SCRATCH disk space, a user can create a Singularity image from an existing public repository (Singularity, Docker, …) by launching the singularity build command:

$ singularity build image-singularity-tensorflow.sif docker://tensorflow/tensorflow:latest-gpu-py3

In this example, the command is used to download an image from the Docker public depository and then build a Singularity image in the SIF format.

The use of containers in SANDBOX form is not authorized on Jean Zay. Nevertheless, it is possible to convert a SANDBOX image to SIF format. You simply need to use the singularity build command:

$ singularity build mon_conteneur.sif mon_conteneur_sandbox/

Comment : As the RAM memory resources are limited to 5GB per user on the front, it is strongly recommended to use the pre/post-processing nodes for the construction or conversion of containers requiring significant resources.

With the singularity run command, you can launch a shell inside a Singularity container. The execution of this command must be done in interactive mode on a computing node (see documentation about CPU and GPU) interactive code execution.

For example, on the CPU partition, you should open a terminal directly on the compute node where your resources are reserved (here, 10 cores) by using the following command:

$ srun --pty --ntasks=1 --cpus-per-task=10 --hint=nomultithread [--other-options] bash

Then, launch the container in shell mode via the following command:

$ singularity shell $SINGULARITY_ALLOWED_DIR/my_container_cpu.sif

On the GPU partition, the manipulation is almost identical after first allocating the proper resources on a GPU computing node (see documentation about GPU interactive code execution). Then, launch the shell container (as described previously for the CPU partition) and use the --nv option in order to take into account the NVIDIA GPU device:

$ singularity shell --nv $SINGULARITY_ALLOWED_DIR/my_contenaier_gpu.sif

To submit an MPI job in batch on Jean Zay, it is necessary to:

• Create a submission script: The following is an example for the default cpu partition (4-GPUs nodes with 40 physical cores), saved in the singularity_mpi.slurm file:

  singularity_mpi.slurm

  #!/bin/bash
  #SBATCH --job-name=SingularityMPI      # name of job
  #SBATCH --ntasks=40                    # total number of MPI processes
  #SBATCH --ntasks-per-node=40           # number of MPI processes per node
  # /!\ Caution, "multithread" in Slurm vocabulary refers to hyperthreading.
  #SBATCH --hint=nomultithread           # 1 MPI process per physical core (no hyperthreading)
  #SBATCH --time=00:10:00                # maximum execution time requested (HH:MM:SS)
  #SBATCH --output=SingularityMPI%j.out  # name of output file
  #SBATCH --error=SingularityMPI%j.out   # name of error file (here, in common with output)
   
  # go into the submission directory
  cd ${SLURM_SUBMIT_DIR}
   
  # clean out the modules loaded in interactive and inherited by default
  module purge
   
  # loading modules
  module load singularity
   
  # echo of launched commands
  set -x
   
  # code execution from the allowed execution space
  # According to the MPI implementation installed in the container, the srun MPI option must be set to --mpi=pmix_v2 or --mpi=pmix_v3
  srun --mpi=pmix_v2 singularity exec $SINGULARITY_ALLOWED_DIR/my-container_CPU.sif ./exec_mpi

• Submit this script via the sbatch command:

  $ sbatch singularity_mpi.slurm

To submit a GPU job in batch on Jean Zay, it is necessary to:

• Create a submission script. The following is an example for the default gpu partition (4-GPUs nodes with 40 physical cores), saved in the singularity_gpu.slurm file:

  singularity_gpu.slurm

  #!/bin/bash
  #SBATCH --job-name=SingularityGPU      # name of job
  ##SBATCH --partition=gpu_p2            # uncomment for gpu_p2 partition gpu_p2
  #SBATCH --ntasks=1                     # total number of MPI tasks (= number of GPUs here)
  #SBATCH --gres=gpu:1                   # number of GPUs per node (1/4 of GPUs)
  #SBATCH --cpus-per-task=10             # number of cores per task (1/4 of the 4-GPUs node here)
  ##SBATCH --cpus-per-task=3             # number of cores per task (with gpu_p2: 1/8 of the 8-GPUs node)
  # /!\ Caution, "multithread" in Slurm vocabulary refers to hyperthreading.
  #SBATCH --hint=nomultithread           # hyperthreading deactivated
  #SBATCH --time=00:10:00                # maximum execution time requested (HH:MM:SS)
  #SBATCH --output=SingularityGPU%j.out  # name of output file
  #SBATCH --error=SingularityGPU%j.out   # name of error file (here, in common with the output file)
   
  # go into the submission directory
  cd ${SLURM_SUBMIT_DIR}
   
  # cleans out modules loaded in interactive and inherited by default
  module purge
   
  # loading singularity module
  module load singularity
   
  # echo of launched commands
  set -x
   
  # code execution from the allowed execution space by using the --nv option in order to take into account the NVIDIA GPU device
  srun singularity exec --nv $SINGULARITY_ALLOWED_DIR/my-container_GPU.sif python ./my_model.py

• Submit this script via the sbatch command:

  $ sbatch singularity_gpu.slurm

• Executing a Singularity image from a front-end is prohibited. Only the interactive and batch modes are authorized.
• The allowed execution space $SINGULARITY_ALLOWED_DIR is restricted to 20 containers and not taken into account in user quotas.
• Creating a private virtual network is not authorized (requires root privileges).
• Executing containers in SANDBOX form is not authorized on Jean Zay.
• Executing containers within the fakeroot mode is not authorized on Jean Zay.
• For a MPI parallel code execution in batch mode from a Singularity image, and according to the MPI implementation installed in the container, the srun MPI option must be set to --mpi=pmix_v2 or --mpi=pmix_v3.
• With the --bind option, you can mount directories from the host to the container:

  --bind directory_host1:directory_container1,directory_host2:directory_container2, ...

• For an execution on the gpu_p2 partition, it is necessary to specify --partition=gpu_p2 and --cpus-per-task=3.
• As the RAM memory resources are limited to 5GB per user on the front, it is strongly recommended to use the pre/post-processing nodes for the construction or conversion of containers requiring significant resources.

• Performance tests of a Deep Learning training with and without Singularity containerisation

Official documentation for Singularity