IDRIS: GENERAL INTRODUCTION



1. Introduction to IDRIS

The Missions and Objectives of IDRIS

The Institute for Development and Resources in Intensive Scientific Computing (IDRIS), founded in 1993, is a centre of excellence in intensive numerical calculations which serves the research branches of extreme computing, as much on the application aspects (large-scale simulations) as on those linked to research inherent to high performance calculations (calculation infrastructures, resolution methods and associated algorithms, processing of large data volumes, etc.).

IDRIS is the major centre of high performance computing for the French National Centre for Scientific Research (CNRS).  Together with the two other national centres, CINES (the computing centre for the French Ministry of Higher Education and Research) and the Very Large Computing Center (TGCC) of the French Alternative Energies and Atomic Energy Commission (CEA), and coordinated by GENCI (Grand Équipement National de Calcul Intensif), IDRIS participates in the installation of national computer resources for the use of government-funded research which requires extreme computing means.

Management of Scientific Resources

A call for proposals is launched during the fourth trimester of each year for the use of the high computing resources of the three national computing centres (CINES, IDRIS and TGCC).  Under the coordination of GENCI (Grand Équipement National de Calcul Intensif), computing hours are allocated for the following year through a unified procedure. 

The requests for resources are made by using the form called DARI (Demande d'Attribution de Ressources Informatiques) through a site common to the three computing centres.

The IDRIS User Committee

The role of the User Committee is to be a liaison for dialogue with IDRIS so that all the projects for which computer resources have been allocated can be carried out in the best conditions. The committee transmits the observations of all the users on the functionning of the centre and discusses the issues with IDRIS to determine the appropriate adaptions to be made.

For more information, consult the website page correspondent.

IDRIS personnel

Organisation

To fulfill its missions, IDRIS is organised into teams coordinated by the director, in the following manner:

IDRIS Organisation chart

Return to Table of Contents


2. The IDRIS machines

HPC resources

CNRS/IDRIS Infrastructure informatique

Computer Number of cores Memory Peak performance
IBM Blue Gene/Q de l'IDRIS Turing: IBM Blue Gene/Q 98,304 96 TiB 1.258 Pflop/s
IBM x3750M4 de l'IDRIS Ada: IBM x3750M4 10,624 46 TiB 230 Tflop/s
Archives serveur         Local disk space Available space on
    tapes    
Serveur d'archives, solution IBM Ergon: IBM solution + Storagetek SL 85 000 2 Pio 10 Pio

Turing : Introduction

Turing is the massively parallel architecture of IDRIS. This is an IBM Blue Gene/Q machine which is particularly well-balanced (network, I/O, computing power and memory bandwidth) and has a remarkable energy efficiency (2.17 Gflops/W).

Hardware characteristics of Turing :

  • 6 racks, each containing 1,024 compute nodes and each node having 16 cores, for a total of 98,304 cores
  • Cumulated peak performance of 1.258 Pflop/s (42nd in the TOP500 worldwide ranking of November 2014)
  • 96 TiB of total memory (1 GiB per core)
  • GPFS parallel file system (WORKDIR) shared by Ada, Adapp and Turing, with a bandwidth of 50 GiB/s in write and in read.

The codes which run on Turing must have a sufficiently high degree of parallelism (ideally about 512 or more execution cores) and not use more than 1 GiB of memory per core. In production, up to 4 racks of resources may be accessed (or 65,536 cores). For the phases of development or code porting, specific classes are available (from 1 to 64 compute nodes). You may consult the batch limits for Turing at class structures.

For more information concerning the available software or librairies, the architecture, and the usage of the Turing machine resources, see here.

Ada : Introduction

Ada is the IDRIS computer with the most wide-ranging usage. It is composed of large memory SMP nodes (IBM x3750-M4) interconnected by a high-speed InfiniBand network.

Hardware characteristics of Ada :

  • 332 x3750-M4 compute nodes, a quadri-socket node of 4 Intel Sandy Bridge E5-4650 8-core processors at 2.7 GHz, with 32 cores per node
    • 304 nodes with 128 GiB of memory (4 GiB/core)
    • 28 nodes with 256 GiB of memory (8 GiB/core)
  • Cumulated peak performance of 233 Tflop/s
  • InfiniBand FDR10 Mellanox network (2 links per node)
  • GPFS parallel file system (WORKDIR) shared by Ada, Adapp and Turing, with a bandwidth of 50 GiB/s in write and in read.

A large variety of codes can be run on Ada, including those requiring a large amount of memory, whether sequential application, multithreaded codes (OpenMP), or codes (MPI or hybrid) having an average degree of parallelism (from hundreds to a few thousand execution cores). In usage, the resources are accessible for up to 2048 execution cores. You may consult the interactive and batch class limits for Ada at class structures.

For more details concerning the available software or librairies, the architecture or the usage of this machine's resources, see here.

Adapp: Introduction

Adapp is the IDRIS computer dedicated to intensive pre- and post-processing and to managing great masses of data. Adapp must only be used for jobs with the following characteristics:

  • Performing many I/Os (e.g. re-combining files)
  • Requiring a large amount of memory (e.g. mesh generating and partitioning)

This pre-/post-processing Adapp architecture contains only 4 compute nodes, each one having:

  • Four 8-core Intel Westmere processors at 2,67GHz (or 32 cores)
  • 1 TB of memory
  • 8 internal disks of 600 GB

The use of these nodes is subject to the following constraints:

  • Maximum Elapsed time of 20h for both sequential and parallel jobs.
  • Maximum reservation of 32 cores (or 1 compute node).
  • You may request up to 100 GB of memory for a sequential execution or 30 GB per reserved core in parallel classes (MPI, OpenMP or hybrid).

Interactive and batch limits may be consulted at class structures for Ada/Adapp.

Please note that you just have to add the keyword
# @ requirements = (Feature == “PREPOST”)
for your job to be submitted on Adapp. However, it must respect the limits indicated above (Elapsed time, number of cores and memory) in order for it to run.

From Adapp (interactive or batch), the HOME of the Ergon archive server is directly accessible in read/write with the environment variable $ARCHIVE, through the intermediary of a GPFS mount.

For more information concerning the available software or librairies, the architecture and the usage of the Adapp resources, see here.

Ergon: Introduction

Ergon is the IDRIS archive server. It is a hierarchical backup system with file storage on disk (2 PB) and automatic migration on tapes (10 PB). Having a very large capacity, data can be archived for medium and long term throughout the duration of a project.

Ergon is an IBM solution composed of 13 servers, including:

  • 3 front-ends, each with 2 processors (Intel Xeon E5-2650) ⇒ 16 cores @2.6 GHz and 128 GB of memory.
  • 6 GPFS file servers (4 data servers and 2 metadata servers) associated with 2 GSS26 disk enclosures with a bandwidth of 12 GB/s and a total available capacity of 2 PB.
  • 2 TSM HSM servers to manage the robotics and file migration to tapes.

Storage: A StorageTek SL8500 robot equipped with 16 readers and containing 6300 tapes, it offers a capacity of 10 PB. The maximum capacity of the robot is 10,000 tapes.

For more information concerning the architecture and the usage of the Ergon resources, see here.

Return to Table of Contents


3. Requesting allocations of hours on IDRIS machines

Requesting resource hours at IDRIS

Requesting resource hours for IDRIS machines is done via the DARI site which is common to the three national computing centres, CINES, IDRIS and TGCC: www.edari.fr.

The modality of attribution of calculation hours at the national computing centres changed in 2017 in order to equilibrate the allocation sessions and separate them from the civil year. Because 2017 is a transition year for this new modality, only one call for projects has been organised which will simultaneously attribute:

  • The A3 hours allocation, accessible to projects having submitted a request for the A1 allocation (or before), and to new projects: These allocated hours are usable during one year, from 1 November 2017 to 31 October 2018.
  • A2 complementary hours allocation of 6 months (IDRIS and CINES only): These allocated hours will be usable from 1 January 2017 to 30 April 2018.

You may submit a request for one of these two allocations, in function of the machines being applied for and your schedule, on the DARI site (https://www.edari.fr).

Submission of allocation requests dates: Tuesday, 6 June 2017 to Tuesday, 5 September 2017

For more information: note de cadrage

During the course of the year, you still have the possibility of requesting:

  • Supplementary resources (demandes au fil de l'eau) for existing projects which have used up their quota of hours during the year.
  • Preparatory access (accès préparatoire) for projects which do not yet have an existing account at IDRIS, or those which have an account on only one of the computing machines (Turing or Ada) and would like preparatory access on the other machine.
    After a request for preparatory access is approved, the resources accorded are 50 000 hours on Turing and 15 000 hours on Ada. Access is granted for a period of 6 months beginning at the project account(s) opening.

All the necessary information for making these requests is found on the DARI site help page: https://www.edari.fr/aide_en_ligne.html (in French).

Return to Table of Contents


4. How to obtain an account at IDRIS

Account management: opening and closing of accounts

Opening a user account

When a project has been approved and obtained hours on one of the IDRIS computers, it is given an identifying “project number”.
Each user in the project must request the opening of his/her account. There is no automatic or implicit account opening.

For a new project

  • If you don't have an account currently opened at IDRIS:

Each individual person who asks for an account opening for the first time at IDRIS must complete the User Login Creation Form (FCCU).

Attention: In application of the regulatory measures for the protection of the national scientific and technical potential (PPST), the creation of a new account may require a French ministerial (MESR) authorisation. The decision whether to pursue this authorisation is made by the IDRIS Director or the CNRS Defence and Security Officer. In this case, a personal communication will be transmitted to the project manager and the concerned user(s) in order to begin implementation of the required procedure. The processing period at the ministry may take up to ten weeks.

  • If you already have an account at IDRIS for another project:

If all your information concerning this account remains valid (laboratory, e-mail address, telephone numbers, etc.), all you have to do is submit an Administration Form for Login Accounts (FGC), completing only the section: “Opening of a supplementary user account for an existing project”.

For a project renewal

Accounts existing on previously accessible computing machines are automatically carried over from one allocation session (project call) to the next.

  • If your account is open and the project has been granted hours in the following allocation session on the same machines, you do not need to re-apply.
  • If an attribution of hours is granted on a computing machine to which the renewed project did NOT have access during the previous allocation, all the users of this project must request the opening of an account on this new machine by using the FGC form (section: “Opening of a supplementary user account for an existing project”).

Moreover, the FGC form may also be used to make necessary modifications on an existing account (addition/removal of IP addresses or changes in postal address, phone number, employer, etc.).

How to transmit forms to IDRIS

All forms must be transmitted to IDRIS by e-mail to the following address: .

Closure of a user account

An account can only exist as “open” or “closed”:

  • Open. In this case, it is possible to:
    • Submit pre- and post-processing jobs on Adapp.
    • Submit bonus jobs on the compute machines if the project still has remaining bonus hours (cf. the last lines of the 'cpt' command output).
    • Submit jobs on the compute machines if the project's current hours allocation has not been exhausted (cf. 'cpt' command output).
  • Closed. In this case, the user can no longer connect. An e-mail notification is sent to the project manager and to the user when the account is being closed.

Attention: After an account is closed, the files can be deleted at any time by IDRIS.

Account closure per request of project manager

A project manager request for account closure is done via the FGC form.
It is the responsibility of the project manager to close the account of a project member who is no longer working on the subject matter of the project or who, having changed employer, is no longer paid by a French research organisation (cf. GENCI hours allocation, section 3, “Conditions d’éligibilité” Note de Cadrage).

ATTENTION :

  • The request for account closure concerns all the machines for which this account is open.
  • Account closure will result in the destruction of all the account files by IDRIS after an undefined time delay.
  • Only the project manager has the possibility of requesting the copying of these files to another account (to be indicated on the FGC form).
    It is indispensable, therefore, to verify that the available space is sufficient for the copies (project disk quotas on the servers).
  • Whether the files are copied to another account or not, only the project manager may request that the account files be immediately destroyed during the account closure to, for example, liberate the disk quotas (request “immediate purge” on the FGC form).

Account closure by IDRIS

Account closure of an unrenewed project

When a GENCI project is not renewed, the following procedure is applied on the date of project expiration (regardless of the project's lifespan):

  • DARI or bonus hours are no longer available. The project accounts can no longer submit jobs on the computing machines (except Adapp); the accounts, however, remain open.
  • Three months later, all the project accounts on all the servers will be closed.

File recovery, by transferring the files to a local laboratory machine, is the responsibility of the user during the three months which follow the end of an unrenewed project. Example: For an unrenewed project finishing in November, no hours are available on 1 December and the accounts are permanently closed in February. This same procedure applies to projects which are not renewed in May.

Account closure following expiry of the ministerial access authorisation for IDRIS computer resources

The first notification of impending expiry is sent to the user 90 days before the expiry date and a second notification is sent 70 days before the expiry. The account is closed on the expiry date. To avoid account closure, the user is advised to submit a newly completed FCCU form as soon as receiving the first notification so that IDRIS may begin processing a prolongation request.

How do I consult my consumption of hours ?

We advise you to use the Extranet interface. Complete information can be found on this page.

What should I do when I will soon use up all my computing hours?

It is possible to make a request for supplementary hours in order to continue projects which have used up all their allocated hours. This may be done anytime during the year. For more information, consult the following page.

Is it possible to know machine availability?

You may obtain this information by connecting on Adapp. The scheduled times for the machines to be stopped and their availability in real time are displayed at the top of the screen just after the connection.

If there is an important incident, a message will also be displayed on an IDRIS website page which is available without a password.

How do I recover files which I accidentally deleted?

The procedure for file recovery is the same on Adapp, Turing and Ada. Therefore, you simply follow the file recovery procedure on the machine where you lost or deleted your files.

Is it possible to ask IDRIS to transfer files from one user account to another user account?

Only the project manager may make this request. The request must be made by a signed fax or postal mail and must indicate the following:

  • The name(s) of the concerned machine(s)
  • The two login accounts (former and new)
  • The list of files and/or directories to be transferred
  • Following this, the project manager may request closure of the former account: Account management: opening and closing of accounts.

Recuperation of files on an external support

It is no longer possible to recuperate files on an external support (i.e. portable disk drives).

Turing, Ada, Adapp : Access and shells

Management of the shell environment

What shells are available on the IDRIS machines?

The Bourne-Again Shell (bash) and the TC Shell (tcsh) are the two command interpreters installed on the IDRIS machines. The Bourne-Again Shell (bash) is an important evolution of the former Bourne shell (sh) and provides advanced functionalities. Therefore, using bash is highly recommended and it is the default setting on the IDRIS machines.

What environment file is executed during the launching of a login session in bash?

The environment file is .bash_profile and it must be found in your HOME. The environment variables must be put by the user into this file which is automatically executed at each interactive session. Any aliases and user-defined shell functions are put in the .bashrc file which is run at the launching of each sub-shell. To define the aliases in the interactive shell, the user must execute .bashrc in the .bash_profile.

Attention: Overwriting the PATH variable inevitably creates problems. For this reason, it is always advised to conserve the PATH which is provided by the machine. If you wish to add a research directory for the execution of local commands to be used during your future sessions, you must procede as follows in your .bash_profile file:

 export PATH=$PATH:repertoire_a_ajouter 

How to define a convivial environment in bash

The Bash shell proposes two edition modes, emacs and vi. The default mode is emacs. If you prefer the vi mode , you may choose this (or change back to the emacs mode) by using the set command:

set -o vi # to be in vi mode
set -o emacs # to be in emacs mode

Place this command in your .bash_profile environment file.

You may edit the command line by using the editing commands of the chosen editor (emacs or vi). For example, to go on the first character of the command, type: Ctrl-a if in emacs mode, or Esc-0 if in vi mode.

To edit the last command which you have launched, it is the same as going back up a line in your editor (Ctrl-P in emacs mode, Esc-k in vi mode). (See man bash for more information about the edition modes.)

You may use filename completion to avoid typing the name of the file. If the file is present in the directory, you just need to type the first letters of the name, then Esc-Esc if in emacs mode, or Esc- in vi mode.

Return to Table of Contents


7. The disk spaces

Disk Quotas

Principles

Quotas guarantee fair access to disk resources. They prevent the situation where a group of users could consume all of the disk space, thereby preventing other users from working. At IDRIS, quotas limit both the quantity of disk space and the number of files (in inodes) of each project

When group quotas are reached, no more files can be created: Doing this could disturb the jobs being run by your group. Attention: Modification of a file after quotas have been reached may result in the file being erased.

How to know your current disk space consumption and quotas

The command quota_u allows you to obtain the current consumption and quotas for the $HOME. The option -w allows you to obtain the same information for $WORKDIR. The $TMPDIR space is not subjected to quotas.

  • The soft quota represents your disk space allocation. When this limit has been reached, you have a period of one week to decrease the disk occupation to within the limit. If disk occupation is not reduced during this period, no group member can create any more files.
  • The hard quota is the disk space limit which the system imposes. If this limit is reached, it is no longer possible to create any files.
  • The quota_u command displays the consumption per user. This consumption includes all the files belonging to a user, even if they are not located in his/her personal spaces. It takes into account files created in spaces shared by the group ($COMMONDIR) or located in personal spaces of other users.
  • Information from the quota_u command is not updated in real time.

When quotas are exceeded

When a group has exceeded the quota on Ergon, an e-mail is sent to each member of the group. On the other machines, however, no warning e-mail is sent; nevertheless, you are informed by error messages when you manipulate files in the concerned disk space (“disk quota exceeded”).

In case of blockage, it is necessary to delete files or move them to another space such as the $WORKDIR or to the Ergon archiving machine if appropriate. The project manager (or designated replacement) can request a quota increase on the IDRIS extranet site.

1) It is necessary to limit the number of files in order to guarantee an optimal usage of the file systems.

Ergon: The HOME disk space

The HOME

The HOME directory is meant for the long-term storage of data files. These files have a lifespan of one year (either a year after their creation date in HOME or their last access date). Technically, the HOME filesystem corresponds to a hard drive cache and a tape library infrastructure where files are progressively migrated to tapes.

  • This is a permanent space; maximum file size is 500 GB.
  • This file system is directly accessible via ssh. It is also accessible from the compute machines via the ''mfget/mfput'' commands and from the pre/post-processing machine Adapp via the $ARCHIVE environment variable.
  • It is subject to group quotas (limiting both the disk space and the number of files for the group). The IDRIS command quota_u allows consulting these quotas.
  • When using the du (disk usage) unix command on Ergon, you must also use the --apparent-size option to take into account the files migrated to tape.
  • The HOME space is not backed up. However, you may use the ''mfdupli'' command to duplicate a file with the assurance that the two copies will be migrated to two different tapes (the file copy is visible in the DUPLI directory).

Ada, Adapp, Turing: Disk spaces

Three distinct disk spaces (HOME, WORKDIR and TMPDIR) are accessible to users on the Ada, Adapp and Turing computers. A fourth disk space, the ARCHIVE space, is only accessible on Adapp. Each space has specific characteristics adapted to its usage which are described on this page. The paths to access these spaces are stocked in 4 variables of the shell environment: $HOME, $WORKDIR, $TMPDIR et $ARCHIVE.

The HOME

$HOME : This is the home directory during an interactive connection. This space is intended for frequently-used small-sized files such as the shell environment files, the tools, and potentially the sources and libraries of limited size (in space and in number of files). The characteristics of the HOME are:

* A permanent space.
* Backed up daily by the [[web:eng:docs:tina:tina-eng|TiNa software application]].
* Shared by the Adapp and Ada machines.
* Accessible in interactive or batch jobs.
* It is the user's home directory when beginning an interactive connection.  It can also be accessed through the ''$HOME'' variable: \\ \\ <code>$ cd $HOME</code>
* Submitted to [[web:eng:su:shared:quota-eng|group quotas]] which are intentionally rather low : 4GB by default. The IDRIS command ''quota_u'' allows you to see the real situation of your disk occupation and that of each of your group members.

  • Intended to receive small-sized files, the block size is 256 KiB (262 KB) (command stat -f $HOME).

The WORKDIR

$WORKDIR : This is a permanent work and storage space which is usable in batch. In this space, we generally store large-sized files which are used to run batch jobs: data files, executable files, result or restart files, submission scripts and very large source files. The characteristics of WORKDIR are:

  • A permanent space.
  • Not backed up.
  • Common to the 3 machines: Adapp, Ada and Turing.
  • Accessible in interactive or in batch jobs.
  • Composed of 2 sections:
    • A section in which each user has an individual part; it is accessed with the command:

      $ cd $WORKDIR
    • A section common to the UNIX group to which the user belongs. The files to be shared by all the group members can be placed here; it is accessed with the command:

      $ cd $COMMONDIR
  • Submitted to group quotas : 1 TiB (1.1 TB), by default. The IDRIS command quota_u -w allows you to see the real situation of your disk occupation and that of each of your group members.
  • The total WORKDIR size is 932 TiB (1024 TB).
  • Intended to receive large-sized files, the block size is 4 MiB (4.2 MB) (command stat -f $WORKDIR).
  • The WORKDIR is a GPFS disk space for which the bandwidth (about 50 GiB/s in read/write) is shared between the Adapp, Ada and Turing machines. It can occasionally be saturated because of exceptionally intensive usage.

Usage recommendations:

  • Because the WORKDIR is not backed up, the files are not protected from the risk of accidental manual destruction (rm) or a disk failure. Therefore, it is necessary to regularly save the sensitive or important files in the Ergon archive server.

Attention :

  • Since batch jobs can run in the WORKDIR, the files are directly accessible in read/write (permanent space) and do not need to be explicitly copied. However, because several of your jobs can be run at the same time, you must create a unique execution directory for each of your jobs. In addition, the disk space is submitted to group quotas and your job execution can suddenly stop if the quotas are reached. Therefore, you must be aware of both your own activity in this disk space and that of your colleagues. For these reasons, you may prefer running your batch jobs in the TMPDIR.

The TMPDIR

$TMPDIR : An execution directory for batch jobs. The following are the characteristics of the TMPDIR:

  • TMPDIR is a temporary directory.
  • It is only accessible in batch jobs by using the $TMPDIR variable.
  • It is automatically created when a batch job begins and is, therefore, unique to each batch job.
  • It is automatically destroyed at the end of this job: You must, therefore, copy the important files on a permanent disk space (WORKDIR, for example) before the end of the job.
  • TMPDIR is not submitted to group quotas, as is HOME or WORKDIR. However, some security quotas are put in place to avoid the situation where a user could unintentionally completely fill up all of the disk space because of an accidental usage error.
  • Total TMPDIR size is 466 TiB (512,3 TB).
  • Intended to receive large-sized files, the block size is identical to that of the WORKDIR: 4 MiB (4,2 MB) (command stat -f $TMPDIR only works in batch).

Usage recommendations:

Examples of batch jobs using the TMPDIR can be found in the “Code execution/control” documentation (Ada here and Turing here). General advice for using the TMPDIR:

  • For each execution, we assume that the input files necessary for the execution (restart or executable) have previously been stored on a permanent file system (HOME ou WORKDIR). If this is not the case, the first step is to use the archive class and the principle of multi-step jobs (example on Ada here and on Turing here) to copy the necessary files into the WORKDIR by using the command mfget.
  • At the beginning of each batch job execution, we advise you to be in the TMPDIR.
  • Copy the necessary files from WORKDIR into TMPDIR using the command cp.
  • Launch the execution in the TMPDIR.
  • Before the batch job finishes, as a last step, you must:
    • Backup save the significant files (if they are used regularly or might be post-processed) in a permanent file system (HOME or WORKDIR) by using the command cp.
    • Archive files for a longer time period by saving them on the Ergon archive server, using the command mfput in the archive class of your batch job (example for Ada here or for Turing here).

Comments:

  • As the performance in read/write is identical for WORKDIR and TMPDIR, you can avoid making copies between these two directories if the code is able to read or write the files directly into the HOME or the WORKDIR;
  • TMPDIR, like WORKDIR, is a GPFS disk space whose bandwidth (about 50 GiB/s in read/write) is shared by the Adapp, Ada and Turing machines. In consequence, the input/output performance can vary as it can be slowed down in the case of exceptionally high-volume usage.

The ARCHIVE space (from Adapp only)

The ARCHIVE environment variable on Adapp is a link to the HOME of the Ergon archive server. This space is directly accessible in read/write from Adapp, the pre- and post-processing machine, through the intermediary of a GPFS mount.

$ ls -l $ARCHIVE

It is only accessible from the Adapp nodes (interactive or batch). It is not accessible from the Ada or Turing nodes.

HOME and WORKDIR: Data security

To improve the protection of your data stored in your HOME and WORKDIR spaces, we recommend that you respect the security policy put in place at IDRIS.

Summary of disk space characteristics

Note: The last column, entitled ARCHIVE, only concerns Adapp and Ergon.

HOME WORKDIR TMPDIR ARCHIVE
Life span permanent permanent duration of batch job permanent
Shared spaces common to Adapp and Ada (HOME is local on Turing) common to Adapp, Ada and Turing $ARCHIVE on Adapp, $HOME on Ergon
Backup saved yes no no no
Automatic deletion no no yes, at end of batch job no
Access in interactive yes yes no ($TMPDIR is not defined) yes from Adapp
Access in batch yes yes yes yes from Adapp
Block size 256 KiB (262 KB) 4 Mib (4.2 MB) 4 MiB (4.2 MB) 16 MiB (16.8 MB)
Group quotas quota_u : 5 GiB (5.4 GB) by default quota_u -w 1 TiB (1,1 To) by default none quota_u : 1 TiB (1.1 TB) on Ergon

Ergon: The disk space quotas, the quota_u command

The general principle of the disk space quotas is explained here. The specifics of the Ergon quotas are given below.

On Ergon :

  • The $HOME storage space is submitted to a group quota both in number of files (15000) and in volume: 1 TiB (1.1 TB).
  • There is a maximum time period of 14 days (depending on the extent you have surpassed the quota) for the group to return within the quota limits : This is the grace period.
  • During the grace period, it is possible on Ergon to exceed the quota by a small amount, so that an mfput command is not blocked during your simulations.

The quota_u command

  • The Ergon quota_u command allows you to obtain the current disk space consumption (yours and for each login of your group) as well as the quota limits in the $HOME space (volume and number of inodes).
  • If the quota has been exceeded, the grace period (in number of days) is indicated in the Timeleft column.
  • The Ergon quota_u command is based on the du (disk usage) unix command using the --apparent-size option in order to take into account the files migrated to tape.

Ergon: Commands related to the Ergon archive server

On the Ergon archive server

  • mfdupli: secure duplication of a file or directory
  • mfret : modification of file expiry dates
  • quota_u : consumption and quota limits (volume and number of inodes)
  • bbftp : file transfers to a machine exterior to IDRIS
  • mfdods : dods_ls, dods_cp, dods_rm

On Adapp

  • mfls : list of contents for the Ergon HOME space.
  • mfget : file transfers from the Ergon archive server to Adapp.
  • mfput : file transfers from Adapp to the Ergon archive server.
  • mfdupli : secure duplication of an Ergon file or directory.
  • mfret : modification of expiry dates for files in the Ergon HOME.

On the Ada and Turing machines

  • mfls : list of contents for the Ergon HOME space.
  • mfget : file transfers from the Ergon archive server to Ada or Turing.
  • mfput : file transfers from Ada or Turing to the Ergon archive server.

Return to Table of Contents


8. Commands for file transfers

Ergon : The mfget and mfput transfer commands

The mfget and mfput commands were developed by IDRIS to secure the transfers between the IDRIS compute machines and the Ergon archive server and to optimise the transfer speed. The following shows the basic usage for a login rlab001 from a computing server, machine_i:

  • In interactive :
    • Copying a file fic_ergon from Ergon to the computing machine_i, the file becoming fic_calcul :
      $ mfget fic_ergon fic_calcul
    • Copying a file fic_calcul from the computing machine_i to Ergon, the file becoming fic_ergon :
      $ mfput fic_calcul fic_ergon
  • In batch, it is recommended to use the archive class to transfer a group of files between the computing machines and the Ergon archive server:
    • In batch, example of using the mfget command in the archive class:
...           
# @ job_type = serial
# @ class = archive
...       
# Copying the file d1 from the Ergon $HOME to the computing machine WORKDIR        
mfget d1 $WORKDIR        

# Copying the file data2 from the Ergon $HOME to the computing machine $WORKDIR under the name d2       
mfget data2 $WORKDIR/d2        

# Copying the file data3 from the Ergon $HOME subdirectory REP
# to the computing machine $WORKDIR under the name d3       
mfget REP/data3 $WORKDIR/d3        
  • In batch, example of using the mfput command in the archive class:
...
# @ job_type = serial
# @ class = archive
... 

cd $WORKDIR       

# Copying the file r1 from the computing machine to the Ergon $HOME
mfput r1 r1
        
# Copying the file r2 from the computing machine to the Ergon $HOME subdirectory REP under the name result2       
mfput r2 REP/result2        
...

To have more information about the file transfer options and capacities of these commands, we advise you to consult the manuals available on line from the computing servers: man mfget, man mfput.

File transfers using the bbftp command

To transfer large-sized files from IDRIS to your laboratory, we advise you to use BBFTP which is a software optimised for transferring files.

All the information for using the bbftp command is found HERE.

Return to Table of Contents


9. The module command

Ada, Adapp: User instructions for the module command

Introduction

The module command is a means of responding most effectively to your specific needs. That is, to set the environment variables such as compilers, mathematical libraries and tools without you having to look for them. It enhances the environment according to what you want to use.

Syntax

module (avail [product] | load product[/version] | list | switch product/version1 product/version2 | display product[/version] …)

  • ''avail'': lists the different products available and their versions
  • ''load'': loads the product in its default version (noted: default), if a version is not specified
  • ''list'': lists the different products loaded and their versions
  • ''switch'': changes the version of a product already loaded

product represents the choice of:

  • a work mode
  • a compilor
  • a library
  • an application or a tool

version represents the different versions of the same product, and can include:

  • default: version by default. This is taken if you do not specify a version; the default version is generally the best adapted.
  • a number: complete number of the version. 

List of available products

Are you looking for a specific product such as a particular version of a library? Do we have it for your use? To find the response to this question, use the command module avail:

  $ module avail

  --------------------------- /smplocal/pub/Modules/IDRIS/modulefiles/environnement ---------------------------
  compilerwrappers/no           modules/1.147
  compilerwrappers/yes(default) modules/3.2.10(default)
  
  --------------------------- /smplocal/pub/Modules/IDRIS/modulefiles/bibliotheques ---------------------------
  arpack/96(default)               lapack/10.3.6(default)           petsc/real-hypre/3.3-p5
  blas/10.3.6(default)             lapack95/10.3.6(default)         petsc/real-mumps/3.1-p8(default)
  blas95/10.3.6(default)           mumps/4.10.0(default)            petsc/real-mumps/3.3-p5
  cmor/2.8.1(default)              nag/23(default)                  phdf5/1.8.9(default)
  fftw/2.1.5                       ncar/6.1(default)                pnetcdf/1.1.1
  fftw/3.2.2                       netcdf/4.1.3                     pnetcdf/1.3.1(default)
  fftw/3.3.2(default)              netcdf/mpi/4.1.3                 scalapack/10.3.6(default)
  fftw/3.3.3                       netcdf/seq/4.1.3(default)        scotch/6.0.0(default)
  hdf5/1.8.9(default)              p3dfft/2.5.1(default)            udunits/2.1.24(default)
  hdf5/mpi/1.8.9                   parmetis/3.2.0(default)          uuid/1.6.2(default)
  hdf5/seq/1.8.9                   parpack/96(default)              vtk/5.10.1(default)
  hypre/2.9.0b(default)            petsc/3.1-p8
  
  --------------------------- /smplocal/pub/Modules/IDRIS/modulefiles/applications ----------------------------
  abinit/7.0.5(default)        espresso/4.3.2               namd/2.8
  adf/2012.01c(default)        espresso/5.0.1(default)      namd/2.9(default)
  adf/2013.01.r36703           espresso/5.0.2               nwchem/6.1.1(default)
  avbp/avbp(default)           gaussian/g03_D02             siesta/2.0.2
  cdo/1.5.9(default)           gaussian/g09_A02(default)    siesta/3.1(default)
  cp2k/2.3_12343(default)      gromacs/4.5.5(default)       siesta/3.1-pl20
  cp2k/2.4_12578               gromacsplumed/4.5.5(default) vasp/4.6.35
  cpmd/3.13.2                  lammps/2012.10.10(default)   vasp/5.2.12(default)
  cpmd/3.15.3(default)         molcas/7.8(default)          vasp/5.2.2
  espresso/4.2.1               molpro/2010.1(default)
  
  ------------------------------ /smplocal/pub/Modules/IDRIS/modulefiles/outils -------------------------------
  cmake/2.8.10.2(default)   idl/8.2(default)          papi/5.1.0.2(default)     python/3.3.0
  ferret/6.84(default)      nco/4.2.3(default)        paraview/3.98(default)    totalview/8.11.0(default)
  fpmpi2/2.2(default)       ncview/2.1.2(default)     python/2.7.3(default)     xmgrace/5.1.23(default)

This list is permanently evolving. (The above list is dated April 2013.)

For more information, consult the user instructions for the module command in our website pages for the Turing and Ada machines: The module command on Turing or The module command on Ada.

Return to Table of Contents


10. How to submit a calculation

Interactive and batch

You have two possible ways of working: in interactive and in batch.

You need to respect the maximum limits for each of these 2 modes for the elapsed (or clock) time and the memory. These limits were set by IDRIS in order to optimally manage the computer resources. You will find further information concerning these limits by typing the news class command on the machine in which you are interested or by consulting the pages on our website concerning the machine(s) you are using: Turing class structure or class structure for Ada and Adapp

Working in interactive

In general, interactive is used for file management (creation, copies, archiving, back-ups, compilation, …). One of the first things you will do is edit a program source and then compile and execute it. All these operations can be carried out directly on the compute machines by using the designated commands for each of the machines. All the local machines which are registered in our filters can access any of the front-ends directly (on Turing, Ada, and Adapp) with the ssh command.

In interactive:

  • You have access to the front-ends.
  • You do not have access to the compute nodes.

Of course, interactive sessions will also be used to prepare batch jobs.

Working in batch

There are several reasons to work in batch mode:

  • The limits in batch for elapsed/clock time and memory are much higher than in interactive.
  • After submitting your batch job, it is still possible to close your interactive session without any risk to the batch job execution.
  • The resources are well managed as batch jobs are distributed on the machines in function of the resources requested (a job requiring high consumption of resources will be run during off-peak hours such as at night and weekends).
  • Batch processing is also used for your jobs on the Adapp pre-/post-processing machine.
  • Batch is the only mode used on Turing.

At IDRIS, we use LoadLeveler software to manage batch job scheduling on the IBM compute machines (Ada, Turing) and on the pre-/post-processing machine (Adapp). LoadLeveler schedules the jobs according to the amount of resources requested (memory, elapsed/clock time, files) and the number of active jobs at a given moment (the number of jobs for each user and the total number of jobs).

There are 2 basic steps for working in batch: creation and submission.

Creation: Consists of writing a file with the submission directives (options) and all the commands which you want to execute. You must begin wih the submission directives, such as:

  • Job name
  • Elapsed/clock time limit for the totality of the job
  • Maximum limit for the memory occupied for each job process
  • Number of processes (for MPI and/or OpenMP)

After the submission directives are entered, it is recommended to enter the commands in the following order:

  • Go into the TMPDIR (cd $TMPDIR).
  • Copy the input files which are necessary for the execution into the TMPDIR (from the HOME or the WORKDIR) by using the cp command.
  • Launch the execution.
  • Copy the result files which you wish to save into the HOME or the WORDIR (from the TMPDIR) by using the cp command.

Submission: To submit a job (or script), you must use the following command. (To have more details about the command, you may consult the manual on the machines via the man command.)

$llsubmit mon-job

Your job will be placed in a batch class according to the values written in the submission directives (see news class on the machine which interests you). We advise you to set the parameters as accurately as possible (concerning the elapsed/clock time and memory) both to avoid reserving resources which will remain unused and to receive the job results as rapidly as possible.

Comments :

  • Batch mode does not allow the user to intervene during the execution of the job commands. Therefore, file transfers must be carried out without needing to enter a password. (You can cancel the job execution only.)
  • Batch classes on Ada and Adapp share the same scheduler. If you want to run a job on the pre-/post-processing machine (Adapp), you must enter the following LoadLeveler keyword in the submission directives. (If this line is not in the submission directives, the job will be run on Ada.)
# @ requirements = (Feature == "prepost")


Detailed information is available on the IDRIS website www.idris.fr/eng for each of the IDRIS machines (Turing, Ada, Adapp) in the section called “Documentation” (code execution/control, compilation, etc.).

If you need further assistance, please contact the IDIRS User Support Team.

Return to Table of Contents


11. Training courses offered at IDRIS

IDRIS training courses

IDRIS provides training courses destined to users of scientific computing. The courses are available to users of IDRIS resources as well as to those from academic and industrial domains. Most IDRIS courses are included in the CNRS continuing education catalogue CNRS Formation Entreprises, allowing enrollment from both the private and public sectors.

The training courses provided by IDRIS principally address the parallelization paradigms of MPI, OpenMP and hybrid MPI/OpenMP, the keystones for using today's supercomputers. In addition, courses are given on the Fortran and C scientific programming languages. A new training course was introduced in 2014 on the subject of large scale debugging (detection of progamming errors and fine-tuning of applications running on a large number of compute cores).

A catalogue of scheduled IDRIS training courses, regularly updated, is available on our web server: https://cours.idris.fr

Enrollment is free of charge for users affiliated with the CNRS (France's National Centre for Scientific Research) or with the French national education system. These users may enroll directly on our web server: IDRIS courses. All other users must enroll through CNRS Formation Entreprises.

IDRIS training course materials are available on line here.

Return to Table of Contents


12. IDRIS documentation

  • The Web site : IDRIS maintains a regularly updated website www.idris.fr/eng/, grouping together the totality of our documentation (IDRIS news, machine functioning, etc.).
  • Manufacturer documentation : Access to complete manufacturer documentation concerning the compilers (f90, C and C++), the scientific libraries, message-passing libraries (MPI), etc.
  • The manuals : Access to all the Unix manuals with your user login on any of the IDRIS calculators by using the command man.

Return to Table of Contents


13. User support

Contacting the User Support Team

Please contact the User Support Team for any questions, information or for any problems you may have on the IDRIS machines.

The User Support Team may be contacted directly:

  • By telephone at +33 (0)1 69 35 85 55 or by e-mail at
  • Monday through Thursday, 9:00 a.m. - 6:00 p.m. and on Friday, 9:00 a.m. - 5:30 p.m., without interruption.

Note: During certain holiday periods (e.g. Christmas and summer breaks), the support team staffing hours may be reduced as follows:

  • Monday through Friday, 9:00 a.m. - 12:00 (noon) and 1:30 p.m. - 5:30 p.m.

Outside of regularly scheduled hours, the answering machine of the User Support Team phone will indicate the currently relevant staffing hours (normal or holiday).

Administrative management for IDRIS users

For any problems regarding passwords, account opening, access authorisation, or in sending us the forms for account opening (FFCU) or account management (FGC), you must send an e-mail to: