Software and libraries for artificial intelligence
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.
Description
Artificial intelligence software is installed in conda virtual environments made available through modules. These environments can be enriched - or new versions installed - upon request to the IDRIS user support (assist@idris.fr).
Since support for version 2.7 ended on 01/01/2020, we have only been installing new software versions for Python 3 since that date.
Available environments
The conda environments we install are built around several main libraries (PyTorch, TensorFlow, NeMo, etc.). They are named after them.
These environments are accessible via the module command. To activate the environment you are interested in, you must load the corresponding module using module load. For example, to activate the conda environment containing PyTorch version 2.8.0, you would load:
module load pytorch-gpu/py3/2.8.0Loading pytorch-gpu/py3/2.8.0If no version is specified, a default version will be loaded.
module load pytorch-gpuLoading pytorch-gpu/py3/2.8.0Once the environment is activated, you can list all the Python packages installed in this environment with the following commands:
pip list
conda list
Since modules containing a conda environment (PyTorch, TensorFlow, etc.) perform a conda activate when they are loaded, it is not possible to activate two of these modules at the same time, nor is it possible to simultaneously activate a module and a personal conda environment.
You can, however,override an existing module with a local installation
Installed versions
To see the list of installed versions on Jean Zay for a given environment, you must use the module avail command. For example, for PyTorch:
module avail pytorch----- /lustre/fshomisc/sup/hpe/pub/modules-idris-env4/modulefiles/linux-rhel9-skylake_avx512 -----pytorch-cpu/py3/1.4.0 pytorch-gpu/py3/1.7.0+hvd-0.21.0 pytorch-gpu/py3/2.0.0pytorch-cpu/py3/1.7.1 pytorch-gpu/py3/1.7.1 pytorch-gpu/py3/2.0.0-cuda11.8...When you load one of these products via the module command, you also load several dependencies: OpenMPI, CUDA, NCCL, cuDNN,... The versions of these dependencies are indicated when loading the product.
module load pytorch-gpu/py3/2.8.0Loading pytorch-gpu/py3/2.8.0 Loading requirement: gcc/11.4.1 cuda/12.8.0 nccl/2.27.3-1-cuda cudnn/9.10.2.21-12-cuda [...]idr_module_search command 🔎
The idr_module_search command developed by IDRIS allows you to find out which Jean Zay module is best suited to your needs. It identifies which environments contain certain packages, with or without version specification.
idr_module_search [options] <package_name1> <package_name2> ...| Option | Description |
|---|---|
--soft | Activates soft mode. The script returns a module if at least one of the requested packages is present (instead of all). |
--ignore-version | Ignores the specified versions during the search. Only the package name match is taken into account. |
--arch <architecture> | Limits the search to a specific architecture. Accepted values are cpu, v100, a100 and h100 (case-insensitive). By default, all architectures are considered. |
-f [--file] <path> | Searches for packages listed in a text file (one package per line). This option is suited for "requirements.txt" files. |
-h [--help] | Displays help and exits. |
Usage examples:
- Simple search:
idr_module_search numpy - Search for multiple packages:
idr_module_search numpy pandas matplotlib - Soft mode search to find at least one of the requested packages:
idr_module_search --soft numpy pandas - Search filtered by architecture:
idr_module_search --arch a100 tensorflow - Search with version:
idr_module_search "numpy>=1.20.0". It is possible to provide a specific version or a constraint. - Search from a
requirements.txtdependencies file:idr_module_search -f requirements.txt
idr_module_search --arch h100 numpy pandas matplotlibModule amber/22-AmberTools25-mpi-cuda H100: matplotlib==3.10.6 numpy==2.3.3 pandas==2.3.3Module anaconda-py3/2024.06 H100: matplotlib==3.8.4 numpy==1.26.4 pandas==2.2.2Module massivefold/1.5.0 H100: matplotlib==3.10.0 numpy==2.2.3 pandas==2.2.3...- The command returns nothing when you add a version constraint: make sure you have used quotes (
"numpy>=2.1.0"). - Invalid package name: names must comply with the standard format of pip or conda repositories.
- Package names can contain special characters like
*or+, which are interpreted as regular expressions. - If constraints are provided both in the command line and via a file (e.g.,
idr_module_search "numpy==1.26.4" -f requirements.txt), then the command-line constraints take precedence over those in the file.
General remarks on AI environments
The commands module purge, module switch, and module unload do not work with conda. To deactivate a conda environment and activate another one by loading a different module, you must run the following commands:
conda deactivate
module purge
module load <new_environment>
A conda environment dedicated to the field of atomistic simulation is available via the following command:
module load atomistic_simulation/py3
- To use environments compatible with A100 GPUs, you must first load the following module:
module load arch/a100
- To use environments compatible with H100 GPUs, you must first load the following module:
module load arch/h100
(For more information:Modules compatible with A100 and H100 partitions)
The H100 software ecosystem is relatively recent and has poor or no support for overly old versions of libraries. Consequently, user support does not plan to install any TensorFlow version below 2.17, nor any PyTorch version below 2.3.1 on this partition.