⚠ INFORMATION
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.

Singularity Containers

Singularity is the containerisation solution available on Jean Zay.

It is an open source multi-platform containerisation system developed by Sylabs. One of the main advantages of containers is that they allow the reproducibility of scientific computations without significant porting effort from one system to another. This solution is efficient and well-suited for both high-performance computing and artificial intelligence computing.

Using Singularity containers, users can work in environments of their choice and design, and these complete environments can be easily copied and run on the Jean Zay supercomputer.

For more details, you can consult the official Singularity documentation.

Note

• Docker is not available on Jean Zay for security policy reasons (root access required). However, Docker images can be used in a Singularity environment.
• Podman is not available on Jean Zay for technical reasons (compatibility with Jean Zay's file system).

Singularity on Jean Zay

Usage Recommendations

The Singularity image you will be handling may be large. It is recommended to run Singularity commands (build, pull, run, exec, etc.) from the $WORK or $SCRATCH space to avoid space saturation.

Additionally, since RAM resources are limited to 5GB per user on the login nodes, it is strongly recommended to use the pre/post-processing nodes for building or converting containers that require significant resources.

Module

Singularity is accessible via the module command:

module load singularity

To obtain information about the Singularity module, simply use the following command:

module show singularity

Temporary File System

Once the module is loaded, a temporary directory $SINGULARITY_CACHEDIR is automatically created in your personal disk space $SCRATCH. It will contain the container's file system during its execution.

Important

It is strongly recommended not to modify the environment variable $SINGULARITY_CACHEDIR to avoid exceeding the quotas of other disk spaces and to simplify automatic cleanup.

Execution Environment

Once your container is ready, you must place it in the dedicated directory $SINGULARITY_ALLOWED_DIR so that it can be analysed and authorised for use on the compute nodes.

note

The variable $SINGULARITY_ALLOWED_DIR defines a unique path per user, and this path cannot be modified.

Copying and deleting containers in this space is done via the idrcontmgr command. This command authorises or denies the copying of containers to this space by first verifying whether the container complies with the security constraints defined by IDRIS.

warning

This space $SINGULARITY_ALLOWED_DIR is limited to 20 containers.

A container can be copied to the authorised execution space via the idrcontmgr command as follows:

idrcontmgr cp my-container.sif

Deletion is done in a similar manner:

idrcontmgr rm my-container.sif

This command also allows you to display the contents of the authorised execution space:

idrcontmgr ls

Building a Singularity Image

The modes of building Singularity images that require being a superuser (root) are not available on Jean Zay. Therefore, it is not possible to build a Singularity image from a definition file. For this type of image, users are responsible for creating them on another machine and then transferring them to Jean Zay.

However, from one of Jean Zay's login nodes, it is possible to create a Singularity image from existing repositories (Singularity, Docker, etc.) in the form of a .sif file.

Building an Image from a Public Repository

From the disk space $WORK or $SCRATCH, a user can create a Singularity image from an existing public repository (Singularity, Docker, etc.) by running the singularity build command:

singularity build image-singularity-tensorflow.sif docker://tensorflow/tensorflow:latest-gpu-py3

In this example, the command is used to download an image from the public Docker repository and then build a Singularity image in the .sif format.

Converting a SANDBOX Image to .sif Format

The use of containers in SANDBOX form is not authorised on Jean Zay. However, it is entirely possible to convert a SANDBOX image to the .sif format. Simply use the singularity build command:

singularity build mon_conteneur.sif mon_conteneur_sandbox/

Running a Shell in a Singularity Image

The singularity run command allows you to launch a Singularity container and a shell inside this container. Running this command requires the use of interactive mode on a compute node (see the documentation on interactive execution of a CPU code and GPU).

For example, for the CPU partition, you will need to open a terminal directly on a compute node where resources are reserved for you (here 10 cores) using the following command:

srun --pty --ntasks=1 --cpus-per-task=10 --hint=nomultithread [--other-options] bash

Then launch the container in shell mode via the following command:

singularity shell $SINGULARITY_ALLOWED_DIR/mon_conteneur_cpu.sif

For the GPU partition, the procedure is almost identical by allocating the appropriate resources on a GPU node and then launching the shell container as before, using the --nv option to account for the NVIDIA cards:

singularity shell --nv $SINGULARITY_ALLOWED_DIR/mon_conteneur_gpu.sif

Running a Code in a Singularity Image in Batch Mode

Running a Parallel MPI Code

To submit an MPI batch job on Jean Zay, you need to:

• Create a submission script
  Here is an example for the default CPU partition (40 physical cores), saved in the file singularity_mpi.slurm:

singularity_mpi.slurm

#!/bin/bash
#SBATCH --job-name=SingularityMPI      # nom du job
#SBATCH --ntasks=40                    # Nombre total de processus MPI
#SBATCH --ntasks-per-node=40           # Nombre de processus MPI par noeud
# /!\ Attention, la ligne suivante est trompeuse mais dans le vocabulaire
# de Slurm "multithread" fait bien référence à l'hyperthreading.
#SBATCH --hint=nomultithread           # 1 processus MPI par coeur physique (pas d'hyperthreading)
#SBATCH --time=00:10:00                # Temps d’exécution maximum demande (HH:MM:SS)
#SBATCH --output=SingularityMPI%j.out  # Nom du fichier de sortie
#SBATCH --error=SingularityMPI%j.out   # Nom du fichier d'erreur (ici commun avec la sortie)

# on se place dans le répertoire de soumission
cd ${SLURM_SUBMIT_DIR}

# nettoyage des modules charges en interactif et herites par defaut
module purge

# chargement des modules
module load singularity

# echo des commandes lancées
set -x

# exécution du code depuis l'espace d’exécution autorisé.
# Selon l’implémentation de MPI installée dans le conteneur, on positionnera l'option --mpi=pmix_v2 ou --mpi=pmix_v3
srun --mpi=pmix_v2 singularity exec $SINGULARITY_ALLOWED_DIR/my-container_CPU.sif ./exec_mpi

important

For the execution of a parallel MPI code from a Singularity image, and depending on the MPI implementation installed in the container, the --mpi=pmix_v2 or --mpi=pmix_v3 option should be set at execution.

• Submit this script via the sbatch command

sbatch singularity_mpi.slurm

Running a GPU Code

To submit a GPU batch job on Jean Zay, you need to:

• Create a submission script
  Here is an example for the default GPU partition (4-GPU V100 nodes with 40 physical cores), saved in the file singularity_gpu.slurm:

singularity_gpu.slurm

#!/bin/bash
#SBATCH --job-name=SingularityGPU      # nom du job
##SBATCH --partition=gpu_p2            # de-commente pour la partition gpu_p2
#SBATCH --ntasks=1                     # nombre total de taches (= nombre de GPU ici)
#SBATCH --gres=gpu:1                   # nombre de GPU per nœud (1/4 des GPU)
#SBATCH --cpus-per-task=10             # nombre de coeurs CPU par tache (1/4 du noeud 4-GPU)
##SBATCH --cpus-per-task=3             # nombre de coeurs CPU par tache (pour gpu_p2 : 1/8 du noeud 8-GPU)
# /!\ Attention, "multithread" fait reference à l'hyperthreading dans la terminologie Slurm
#SBATCH --hint=nomultithread           # hyperthreading desactive
#SBATCH --time=00:10:00                # Temps d’exécution maximum demande (HH:MM:SS)
#SBATCH --output=SingularityGPU%j.out  # Nom du fichier de sortie
#SBATCH --error=SingularityGPU%j.out   # Nom du fichier d'erreur (ici commun avec la sortie)

# on se place dans le répertoire de soumission
cd ${SLURM_SUBMIT_DIR}

# nettoyage des modules charges en interactif et herites par defaut
module purge

# chargement des modules
module load singularity

# echo des commandes lancées
set -x

# exécution du code depuis espace d’exécution autorisé avec l'option --nv afin de prendre en compte les cartes NVIDIA
srun singularity exec --nv $SINGULARITY_ALLOWED_DIR/my-container_GPU.sif python ./my_model.py

• Submit this script via the sbatch command

sbatch singularity_gpu.slurm

Remarks

• Network virtualisation types such as ipvlan, macvlan, ptp, bridge, etc., are not authorised (as they require being root). The host network is used.
• Running containers in fakeroot mode is not authorised on Jean Zay.
• The --bind option allows you to mount directories from the host to the container:
  --bind directory_host1:directory_container1,directory_host2:directory_container2
• It is currently not possible to launch Jupyter (Notebook or Lab) and TensorBoard from a Singularity container.