Table des matières

Jean Zay: Software for artificial intelligence

Description

The artificial intelligence software is installed in Anaconda virtual environments.

These environments can be supplemented upon request to the IDRIS support team (assist@idris.fr).

Since support for Python version 2.7 ended on 01/01/2020, we are only installing new software versions for Python 3. Only the old versions are still available in Python 2.

Software installed

The Anaconda environments which we install are built around four principal libraries:

  • TensorFlow
  • PyTorch
  • Caffe
  • MXNet

These environments are accessible via the module command. To display the list of available AI modules: 

$ module avail tensorflow pytorch caffe mxnet

To activate your chosen environment, it is necessary to load the corresponding module. For example, to activate the Anaconda environment containing TensorFlow version 2.5.0, you should load: 

$ module load tensorflow-gpu/py3/2.5.0

Once the environment is activated, you can list all the Python packages installed in this environment by using the following commands: 

$ pip list
$ conda list

Comments:

  • An Anaconda environment dedicated to the domaine of atomistic simulation is available via the following command: 
    $ module load atomistic_simulation/py3

    You can find more information on the page of associated documentation.

  • Only the PyTorch and TensorFlow environments from pytorch-1.11.0 and tensorflow-2.8.0 versions are compatible with the Nvidia A100 SXM4 80 Gb GPUs of the ''gpu_p5'' partition. These 52 nodes having AMD CPUs, you must therefore load cpuarch/amd module before to load PyTorch or TensorFlow module: 
    $ module load cpuarch/amd

    For more informations: Modules compatible with ''gpu_p5'' partition.

Installed versions

To see the list of versions installed on Jean Zay, you must use the module avail command:

  • For TensorFlow 
    $ module avail tensorflow
  • For PyTorch 
    $ module avail pytorch
  • For Caffe 
    $ module avail caffe
  • For MXNet 
    $ module avail mxnet

The names of the products appear in the form: <software>-gpu/<python version>/<product version>. Except for MXNet which have been installed after the end of life of Python 2.
For example, TensorFlow version 2.2.0 for Python 3 will be named tensorflow-gpu/py3/2.2.0. When you load one of these products via the module command, you also load the following dependencies:

  • OpenMPI
  • CUDA
  • NCCL
  • cuDNN

The versions of these dependencies are indicated at the loading of the product. For example : 

$ module load tensorflow-gpu/py3/2.2.0
Loading tensorflow-gpu/py3/2.2.0
  Loading requirement: cuda/10.1.2 nccl/2.5.6-2-cuda cudnn/7.6.5.32-cuda-10.1 intel-compilers/19.0.4 openmpi/4.0.2-cuda

You are also loading the corresponding Python version (py3 or py2). Note: New versions can be installed upon request to the IDRIS support team (assist@idris.fr).

Comment

The module purge, module switchand module unload commands do not function with Anaconda at this time. To change products during the same session, keeping the same Python version, you can use the following conda commands:

  • conda deactivate to leave the virtual environment of the product you are using
  • conda info -e to obtain a list of all the available virtual environments
  • conda activate nom_environnement_virtuel to activate your desired product.

Example : 

$ module load tensorflow-gpu/py3/1.14
 
$ conda deactivate
$ conda info -e
# conda environments:
#
base                  *  /gpfslocalsup/pub/anaconda-py3/2019.03
caffe-gpu-1.0            /gpfslocalsup/pub/anaconda-py3/2019.03/envs/caffe-gpu-1.0
habitat                  /gpfslocalsup/pub/anaconda-py3/2019.03/envs/habitat
pytorch-gpu-1.1          /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.1
pytorch-gpu-1.2.0        /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.2.0
pytorch-gpu-1.3          /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.3
pytorch-gpu-1.3.0+nccl-2.5.6-2     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.3.0+nccl-2.5.6-2
pytorch-gpu-1.3.1        /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.3.1
pytorch-gpu-1.3.1+nccl-2.4.8-1     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.3.1+nccl-2.4.8-1
pytorch-gpu-1.3.1+nccl-2.5.6     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.3.1+nccl-2.5.6
pytorch-gpu-1.3.1+nccl-2.5.6-2     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.3.1+nccl-2.5.6-2
pytorch-gpu-1.4.0        /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.4.0
pytorch-gpu-1.4.0+nccl-2.5.6-2     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.4.0+nccl-2.5.6-2
pytorch-gpu-1.5.0+nccl-2.4.8     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/pytorch-gpu-1.5.0+nccl-2.4.8
tensorflow-gpu-1.12      /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-1.12
tensorflow-gpu-1.13      /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-1.13
tensorflow-gpu-1.14      /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-1.14
tensorflow-gpu-1.14-mpi     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-1.14-mpi
tensorflow-gpu-1.4       /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-1.4
tensorflow-gpu-1.8       /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-1.8
tensorflow-gpu-2.0.0-beta1     /gpfslocalsup/pub/anaconda-py3/2019.03/envs/tensorflow-gpu-2.0.0-beta1
 
$ conda activate tensorflow-gpu-2.0.0-beta1