Ergon: Usage of the Ergon archive server

The utilisation of this archive server supports the current IDRIS policy regarding the storage of results. Accordingly, it is recommended to use this server for archiving purposes and no longer for temporary storage. We advise all users to employ the WORKDIR space (common to the Turing and Ada computers) for all their active data: This space constitutes the work directory of the batch jobs. For a simulation campaign, all the files should be stored in the WORKDIR, even large-sized files (data files, executable files, result files or calculation restarts), without transferring the result files each time to the archive machine and then needing to re-transfer them in the other direction before beginning the next step in the simulation. The WORKDIR space has large group quotas which can be increased by a simple justified request from the project manager (IDRIS Extranet). However, as the WORKDIR space is not backed up, it is nevertheless necessary to regularly save the particularly important or sensitive files in the Ergon archive server by copying them there using the specialised commands, mfget/mfput.

Should I compress my files to stay within my quotas?

You should not, under any circumstances, compress your files if you are intending to archive them on Ergon.

If you lack archiving space on Ergon, the project manager should simply submit a justified request (IDRIS Extranet) to obtain the necessary space.

In fact, the compression software (via gzip/compress) has proven to be very disadvantageous for the exploitation of Ergon for the following reasons:

  • It is less effective than the hardware compression which is systematically implemented on Ergon during the migration onto tape.
  • It consumes CPU resources both at the compression and at the decompression.
  • It generates the creation of new files and the deletion of old ones, causing an important fragmentation of the tapes which, as a result, have numerous 'holes'. The assessment is very negative, therefore, in terms of occupation of space on the tapes.

For these reasons, you should strictly reserve the (tar + gzip) compression for large files which you wish to transfer outside of IDRIS. Of course, there is no particular problem with using tar without compression for the purpose of grouping files and this is, in fact, encouraged.

Information for the user:

  • Each IDRIS user has an account on the Ergon archive server.
  • The password and the Ergon connection shell (either bash or tcsh) are shared by the Ada, Adapp and Turing machines. You just need to change it on one of the machines and it will also be effective on the others.
  • The interactive connection on Ergon is still possible but it is recommended, in general, to work directly from the Adapp machine where all the pre/post-processing tools, commands and libraries are available.
  • No pre-/post-processing software or libraries will be installed on Ergon.
  • The Ergon storage space, called HOME, is subjected to quotas both in the number and volume of files.
  • All the commands related to file management in HOME are available on both Ergon and Adapp. The Ergon HOME space is directly accessible from Adapp with read/write permissions by using the environment variable $ARCHIVE (through the intermediary of a GPFS mount).
  • There is no direct access to the Ergon HOME from the Ada and Turing compute machines. You must explicitly transfer the files from one machine to another by using (by preference) the secured mfget/mfput commands. On the compute machines, it is recommended to uncouple the file transfer phases from/towards the archive server (use of the secured mfget/mfput commands) from the computation steps by using multi-step jobs and the archive class dedicated to the transfer (Turing: Multi-step jobs; Turing: Multi-step jobs with file transfers using Ergon; Ada: Execution of multi-step and cascade jobs).