Flash Info No 2025-03
This page was translated by an AI (LLM) with a cursory human check and is awaiting full review.
Migration of the DSDIR space to Lustre
[English version below]
Hello,
During the maintenance on January 21, 2025, the DSDIR environment variable was modified and now points to a new Lustre storage space: "/lustre/fsmisc/dataset".
All databases and models downloaded to DSDIR since the beginning of Jean Zay have been migrated to this new space. This new space is accessible from all Jean Zay nodes, including the nodes of the new H100 extension.
This new space will now be used for downloading public datasets or models. The old disk spaces "/gpfsdsdir/dataset" and "/gpfsdswork/dataset" are now deprecated and will no longer be accessible as of the end of January.
If you are already using the DSDIR environment variable in your scripts, the data migration will have no impact on your runs. However, if you are using hardcoded paths, you will need to update them. We invite you to take this opportunity to switch to systematic use of the DSDIR variable, as in the following script for example:
''' import os from datasets import load_dataset dataset = load_dataset(os.environ['DSDIR']+"/HuggingFace/wikimedia/structured-wikipedia") '''
As a reminder, DSDIR is a dedicated disk space for storing public databases or models needed for the use of Artificial Intelligence tools. It is visible to all Jean Zay users. For more information: http://www.idris.fr/eng/jean-zay/cpu/jean-zay-cpu-calculateurs-disques-eng.html#the_dsdir
Please do not hesitate to contact the support team at assist@idris.fr if you encounter any issue.
Best regards, The IDRIS support team