Ada: Safe file transfers with the Ergon archive server

The mfget and mfput commands were developed by IDRIS to:

  • Secure transfers between the IDRIS compute machines and the Ergon archive server.
  • Optimise the transfer speed.

mfget - File transfer from Ergon to Ada

Syntax of the mfget command:

  • mfget [-v] [-p] file_name
  • mfget [-v] [-p] ergon_file_name ada_file_name
  • mfget [-v] [-p] ergon1_file_name ergon2_file_name [ergon3 file_name …] ada_directory

    where

    • file_name, ergon_file_name, ergon1_file_name, ergon2_file_name, ergon3_file_name: files to transfer from Ergon
    • ada_file_name: name of the local file on Ada
    • ada_directory: name of the local directory on Ada

Example : To copy the ergon_file located on the Ergon HOME under the ada_file name in the Ada TMPDIR directory:

$ mfget ergon_file $TMPDIR/ada_file

mfput - File transfer from Ada to Ergon

Syntax of the mfput command:

  • mfput [-v] [-p] file_name
  • mfput [-v] [-p] ada_file_name ergon_file_name
  • mfput [-v] [-p] ada1_file_name ada2_file_name [ada3_file_name …] ergon_directory

    where

    • file_name, ada_file_name, ada1_file_name, ada2_file_name, ada3_file_name, …: file names on Ada to transfer to Ergon
    • ergon_file_name: name of distant file on Ergon
    • ergon_directory: name of the target directory on Ergon

Example: To copy the ada_file located on the Ada WORKDIR under the ergon_file name in the Ergon HOME directory:

$ mfput $WORKDIR/ada_file ergon_file

Comments

  • The Ada HOME has a different name than the Ergon HOME. Therefore, you must not code in the following way:
$ mfget $HOME/sub_dir/ergon_file

$ mfput ada_file $HOME/sub_dir/ergon_file

Instead, you must code in this way:

$ mfget sub_dir/ergon_file

$ mfput ada_file sub_dir/ergon_file
  • In case of failure (verbose shell):
    If your login shell on Ergon is the bash or the tcsh and an output on the screen is generated during the execution of your environment files (.bashrc or .tcshrc), then the transfer has failed. The solution consists of modifying your environment files to make them «mute», as follows:
    • In a .tcshrc file:
      if ( { tty -s } ) then 
      # Execute only if you are in an interactive session 
        echo "message" 
      endif
    • In a .bashrc file :
      if $(tty -s)
      then 
      # Execute only if you are in an interactive session
        echo "message" 
      fi 
  • It is preferable to assemble a large number of files via the tar command before transferring a single created archive file, thereby obtaining the efficient transfer of a large file (GBs) instead of a multitude of small files.
  • If there is a large volume of transfers, it is advised to use the dedicated “archive” class in a multi-step job because the elapsed time is invoiced.
  • In case of a network problem between the compute machine and Ergon, the transfer is automatically relaunched three times.
  • In case of a complete transfer failure beginning at the $TMPDIR, an error message indicates the place where the non-transferred files were copied and where they are recoverable.

For more information about the transfer options and capacities of these commands, we advise you to consult their manuals on the compute servers (man mfget, man mfput).

Transferring between two different user accounts

By default, the distant user name (login) used by the mfget/mfput commands on Ergon is the same as the local user name on the compute machine. The commands will, therefore, search for or drop the files in the HOME of the Ergon user. To change this behaviour and recover/copy a file in the HOME of another user on Ergon, you just need to indicate the complete access path of the file. It is necessary, of course, that the UNIX and/or ACL access rights be correctly positionned on the two directories so that read/write access will be possible.
For example, with the following mfget command, the login_1 user will search for the file rep_login_2_ergon/fich_ergon in the HOME of login_2 on Ergon and then copy it into his own WORKDIR on Ada:

$ mfget /path/to/home/login_2/rep_login_2_ergon/fich_ergon $WORKDIR/fich_ada