Table des matières
Python personal environment
IDRIS offers many modules dedicated to AI. The modules are designed for the most efficient use of the Jean Zay intrastructure and the installed libraries (such as cuDNN, Intel MKL, NCCL).
If a module corresponds to your needs but lacks libraries which are indispensable for your codes, you have two solutions:
- Make a request to the user support team (assist@idris.fr) to install the library and specify the concerned library and module. This will also potentially benefit other Jean Zay users.
- Add the missing libraries locally via the
pip
command, by loading one of the proposed modules as an extension to the environment. The advantages and disadvantages of this method are described below.
If what you need is really very different from what is offered by IDRIS, you can create your own conda environment with exactly what is necessary for you. This gives you total control but there are some disadvantages:
- An environment takes up space (particularly in terms of inodes) and could saturate your allocated space. See the following section for tactics to limit this problem.
- The installed codes are not optimised for Jean Zay and are not necessarily going to find core libraries such as cuDNN (even by loading the
cudnn/7.6.5.32-cuda-10.2
). - The user support team will have more difficulty assisting you in case your code shows abnormal conduct.
Some useful commands:
- To know which modules are available:
module avail
, for examplemodule avail python
- To load a module:
module load <module_name>
- To obtain a complete list of associated modules:
module list
(after a module is loaded) - To “unload” all the modules:
module purge
(in case of anomaly, it could be preferable to begin a new session)
General advice before installing libraries locally
The storage directory for packages PyPI is situated in $HOME/.local
by default. Therefore, you could rapidly saturate the disk quota of your $HOME
if you keep this default behaviour. To avoid this, we recommend that you create a target directory in your $WORK
and create a symbolic link in the $HOME
which points to the target directory:
- If the
$HOME/.local
file already exists, you must first move it to the$WORK
:
$ mv $HOME/.local $WORK $ ln -s $WORK/.local $HOME
- If the
$HOME/.local
file does not yet exist, you must first create a directory in the$WORK
:
$ mkdir $WORK/.local $ ln -s $WORK/.local $HOME
Likewise, via conda
, libraries are installed in $HOME/.conda
; it is strongly encouraged, therefore, to proceed in the same way.
- If the
$HOME/.conda
file already exists, you must first move it to the$WORK
:
$ mv $HOME/.conda $WORK $ ln -s $WORK/.conda $HOME
- If the
$HOME/.conda
file does not yet exist, you must first create a directory in the$WORK
:
$ mkdir $WORK/.conda $ ln -s $WORK/.conda $HOME
To know the usage of the $WORK
:
idr_quota_user --space work
gives the usage percentage in GB and inodes but does not provide real-time information (updated every morning).- For more detail, enter
du --inodes --max-depth=1 $WORK
ordu -h --max-depth=1 $WORK
.
Adding libraries locally via pip
Advantages and disadvantages of this method:
- This method is suited for adding libraries to an existing module and avoids needing to install everything.
- In return, however, libraries installed via
pip
will be visible to every module which uses the same version of Python (installation is done in.local/lib/pythonx.y
). - It is necessary, therefore, to be vigilant about possible incompatibilities if you launch calculations with multiple environments (for example,
pytorch-gpu/py3/1.5.0
andpytorch-gpu/py3/1.7.1
are based on the same version of Python (3.7) but not the same version ofnumpy
). - Local installation of a library takes precedence over the version from a module but it is necessary to stay vigilant about the dependencies.
To locally install packages from the PyPI storage, use the --user
and --no-cache-dir
options:
$ pip install --user --no-cache-dir <package>
$ pip uninstall <package>
Other useful commands (for the options, see official documentation):
pip cache purge
(useful if you have forgotten the--no-cache-dir
option)pip list
to see all the accessible packages (from the module or from a local installation)
Creation of a conda environment
You do not need to install Anaconda or Miniconda to create your own environment! You just load one of the proposed Python environments which are found by using the module avail python
command.
The useful commands are:
conda deactivate
to exit from the environment loaded withmodule load
conda create -y -n <name> python=x.y
to create an environment (specify a Python version; otherwise, it is the module default version)conda activate <name>
conda env remove --name <name>
to correctly delete the environmentconda clean -a
to remove unused packages and the cache. Do not be concerned if this appears to try to delete the packages of the system environment (ie. non-local).
Installation of packages is done via conda install
. It could be wise to add -c conda-forge
as this is the channel which contains the latest updates of many scientific packages.
Final advice
$PATH
, $PYTHONPATH
and $CONDA_PREFIX
are configured correctly; it should not be necessary to modify them (which could change the directory precedence order, or result in paths to incompatible packages).
However, if a package installs an executable (this is the case in Horovod and Deepspeed), it will be in .local/bin
. To access this executable on a compute node, it will be necessary to enter export PATH=$WORK/.local/bin:$PATH
(in the Slurm file or, after being connected, on a node).
Alternatively, if you use multiple Jean Zay modules (for example, Tensorflow and Pytorch), which you complete via the “pip” installations, it is possible to redefine the location of locally installed packages by defining the PYTHONUSERBASE
variable: for example, export PYTHONUSERBASE=$WORK/.local_tf240
. This is done before the package installation and before launching the computations (remember to eventually redefine the PATH
if necessary).
If ever you need to compile AI codes using the GPUs (CUDA, cuDNN), intel MKL, etc., do not hesitate to contact the user support team (assist@idris.fr).