Turing: Safe file transfers with the Ergon archive server

The mfget and mfput commands were developed by IDRIS to:

  • Secure transfers between the IDRIS compute machines and the Ergon archive server.
  • Optimise the transfer speed.

mfget - File transfer from Ergon to Turing

Syntax of the mfget command:

  • mfget [-v] [-p] file_name
  • mfget [-v] [-p] ergon_file_name turing_file_name
  • mfget [-v] [-p] ergon1_file_name ergon2_file_name [ergon3 file_name …] turing_directory

    where

    • file_name, ergon_file_name, ergon1_file_name, ergon2_file_name, ergon3_file_name: files to transfer from Ergon
    • turing_file_name: name of the local file on Turing
    • turing_directory: name of the local directory on Turing

Example : To copy the ergon_file located on the Ergon HOME under the turing_file name in the Turing TMPDIR directory:

$ mfget ergon_file $TMPDIR/turing_file

mfput - File transfer from Turing to Ergon

Syntax of the mfput command:

  • mfput [-v] [-p] file_name
  • mfput [-v] [-p] turing_file_name ergon_file_name
  • mfput [-v] [-p] turing1_file_name turing2_file_name [turing3_file_name …] ergon_directory

    where

    • file_name, turing_file_name, turing1_file_name, turing2_file_name, turing3_file_name, …: file names on Turing to transfer to Ergon
    • ergon_file_name: name of distant file on Ergon
    • ergon_directory: name of the target directory on Ergon

Example: To copy the turing_file located on the Turing WORKDIR under the ergon_file name in the Ergon HOME directory:

$ mfput $WORKDIR/turing_file ergon_file

Comments

  • The Turing HOME has a different name than the Ergon HOME. Therefore, you must not code in the following way:
$ mfget $HOME/sub_dir/ergon_file

$ mfput turing_file $HOME/sub_dir/ergon_file

Instead, you must code in this way:

$ mfget sub_dir/ergon_file

$ mfput turing_file sub_dir/ergon_file
  • In case of failure (verbose shell):
    If your login shell on Ergon is the bash or the tcsh and an output on the screen is generated during the execution of your environment files (.bashrc or .tcshrc), then the transfer has failed. The solution consists of modifying your environment files to make them «mute», as follows:
    • In a .tcshrc file:
      if ( { tty -s } ) then 
      # Execute only if you are in an interactive session 
        echo "message" 
      endif
    • In a .bashrc file :
      if $(tty -s)
      then 
      # Execute only if you are in an interactive session
        echo "message" 
      fi 
  • It is preferable to assemble a large number of files via the tar command before transferring a single created archive file, thereby obtaining the efficient transfer of a large file (GBs) instead of a multitude of small files.
  • If there is a large volume of transfers, it is advised to use the dedicated “archive” class in a multi-step job because the elapsed time is invoiced.
  • In case of a network problem between the compute machine and Ergon, the transfer is automatically relaunched three times.
  • In case of a complete transfer failure beginning at the $TMPDIR, an error message indicates the place where the non-transferred files were copied and where they are recoverable.

For more information about the transfer options and capacities of these commands, we advise you to consult their manuals on the compute servers (man mfget, man mfput).

Transferring between two different user accounts

By default, the distant user name (login) used by the mfget/mfput commands on Ergon is the same as the local user name on the compute machine. The commands will, therefore, search for or drop the files in the HOME of the Ergon user. To change this behaviour and recover/copy a file in the HOME of another user on Ergon, you just need to indicate the complete access path of the file. It is necessary, of course, that the UNIX and/or ACL access rights be correctly positionned on the two directories so that read/write access will be possible.
For example, with the following mfget command, the login_1 user will search for the file rep_login_2_ergon/fich_ergon in the HOME of login_2 on Ergon and then copy it into his own WORKDIR on Turing:

$ mfget /path/to/home/login_2/rep_login_2_ergon/fich_ergon $WORKDIR/fich_turing