Jean Zay: Softwares for artificial intelligence

The artificial intelligence softwares are installed in conda virtual environments.

These environments can be supplemented upon request to the IDRIS support team (assist@idris.fr).

Since support for Python version 2.7 ended on 01/01/2020, we are only installing new software versions for Python 3 since.

The conda environments which we install are built around several main libraries (PyTorch, TensorFlow, NeMo, etc). They are named after these.

These environments are accessible via the module command. To display the list of available AI modules:

$ module avail pytorch tensorflow nemo [other]

To activate your chosen environment, it is necessary to load the corresponding module. For example, to activate the conda environment containing TensorFlow version 2.5.0, you should load:

$ module load tensorflow-gpu/py3/2.5.0

Once the environment is activated, you can list all the Python packages installed in this environment by using the following commands:

$ pip list
$ conda list

To list the versions installed on Jean Zay for a given environment (TensorFlow, PyTorch, caffe or MXNet), you must use the module avail command. For example, regarding TensorFlow:

$ module avail tensorflow

The names of the products appear in the form: <software>-gpu/<python version>/<product version>. Except for MXNet which have been installed after the end of life of Python 2.
For example, TensorFlow version 2.2.0 for Python 3 will be named tensorflow-gpu/py3/2.2.0.

When you load one of these products via the module command, you also load several dependencies: OpenMPI, CUDA, NCCL, cuDNN,… The versions of these dependencies are indicated at the loading of the product. For example :

$ module load tensorflow-gpu/py3/2.2.0
Loading tensorflow-gpu/py3/2.2.0
  Loading requirement: cuda/10.1.2 nccl/2.5.6-2-cuda cudnn/7.6.5.32-cuda-10.1 intel-compilers/19.0.4 openmpi/4.0.2-cuda

You are also loading the corresponding Python version (py3 or py2).

Note: New versions can be installed upon request to the IDRIS support team (assist@idris.fr).

The idr_module_search command developed by IDRIS allows you to know which Jean Zay module is best suited to your needs. It allows you to identify which environments contain certain packages, with or without version specification.

idr_module_search [options] <package_name1> <package_name2> ...

• Simple search: idr_module_search numpy
• Search for multiple packages: idr_module_search numpy pandas matplotlib
• Soft mode search to find at least one of the requested packages: idr_module_search --soft numpy pandas
• Architecture-filtered search: idr_module_search --arch a100 tensorflow
• Search with version specification: idr_module_search "numpy>=1.20.0". It is possible to provide a specific version or a constraint.
• Search from a “requirements.txt” file of dependencies: idr_module_search -f requirements.txt

• The command returns nothing when you add a version constraint: make sure you have used quotes (e.g. "numpy>=2").
• Invalid package name: names must follow the standard format of pip or conda repositories.

• Package names can contain special characters like * or + which are interpreted as regular expression patterns.
• If constraints are provided both on the command line and via a file (e.g. idr_module_search "numpy==1.26.4" -f requirements.txt), those on the command line take precedence over those in the file.

• The module purge, module switchand module unload commands do not function with conda. To deactivate a conda environment and activate another one by loading another module, you need to run the following commands:

  conda deactivate
  module purge
  module load <new_environment>

• To use environments compatible with the A100 GPUs, you need to load the following module beforehand:

  module load arch/a100

  For more informations: Modules compatible with ''gpu_p5'' partition.

• To use environments compatible with the H100 GPUs, you need to load the following module beforehand:

  module load arch/h100

  For more informations: Modules compatible with ''gpu_p6'' partition.
  Important: the H100 software ecosystem is recent and does not support old versions of libraries. We do not plan to install TensorFlow versions lower than 2.17 or PyTorch versions lower than 2.3.1 on this partition.

• A conda environment dedicated to the domaine of atomistic simulation is available via the following command:

  $ module load atomistic_simulation/py3

  You can find more information on the page of associated documentation.