Jean Zay: Computing hours accounting and the idracct command

On Jean Zay, computing hours are allocated per project and with accounting differentiation between CPU and GPU hours.

The accounting hours consumed by your jobs are determined on the basis of:

  • The number of reserved physical cores × the elapsed time for a CPU or GPU job.
  • Or/and the number of reserved GPUs × the elapsed time for a GPU job.

Note : for a GPU type job, the number of reserved GPUs taken into account for the job accounting is not necessarily equal to the number of GPUs requested via the Slurm directive #SBATCH --gres=gpu:.... Indeed, an adjustment can be made according to the number of CPUs which is also requested for the job. For example, requesting a single GPU from a node (i.e. 1/4 or 1/8 of the GPUs depending on the type of node) via #SBATCH --gres=gpu:1 and half of the CPUs of the node, leads to the reservation of half of the resources of the node. The job accounting is then calculated on the basis of a reservation of half of the GPUs (so as if you had requested #SBATCH --gres=gpu:2 or #SBATCH --gres=gpu:4 depending on the type of node).

Note that during an execution in exclusive mode (the --exclusive option of the sbatch command or the Slurm directive #SBATCH --exclusive), you reserve all of the compute nodes:

  • 40 physical cores for cpu partitions.
  • 40 physical cores and 4 GPUs for default gpu partition.
  • 24 physical cores and 8 GPUs for gpu_p2 partition.

The accounting will then be on the basis of:

  • The number of reserved nodes × 40 cores × elapsed time for a CPU job.
  • The number of reserved nodes × 4 GPUs × elapsed time for a GPU job in default gpu partition..
  • The number of reserved nodes × 8 GPUs × elapsed time for a GPU job in gpu_p2 partition.

IMPORTANT: All executions requiring one or more compute nodes are automatically run in exclusive mode and thus are invoiced on the basis of the number of full nodes reserved.

You can consult your project allocation by using the idracct command (see section below).

Moreover, we recommend that you read the documentation about the Regulating of computing hours which influences the number and usage of computing hours.

Hours accounting

The hours accounting depends on if your login is attached to a single project or to several projects, and if the projects have CPU or GPU hours or both:

  • If your single project has only CPU or GPU hours, you do not need to specify for which allocation the consumed computing hours should be counted.
  • In contrast, if your single project has both CPU and GPU hours or if your login is attached to 2 or more projects, you must imperatively specify the hours allocation on which your consumed computing hours should be counted. Without this information, your consumed hours could be counted for a different project than the one on which you think you are working.
    It is obligatory, therefore, that you add either the Slurm directive #SBATCH --account=... in the header of the submission script, or the --account=... option to the sbatch command, taking care to specify the right project (my_project) and the type of hours to use (cpu or gpu).
    # For CPU hours
    # As argument of the sbatch, salloc or srun commands:
    $ sbatch ... --account=my_project@cpu job_cpu.slurm
     
    # Or via the SLURM directive in a batch job:
    #SBATCH --account=my_project@cpu
    # For GPU hours
    # As argument of the sbatch, salloc or srun commands:
    $ sbatch ... --account=my_project@gpu job_gpu.slurm
     
    # Or via the SLURM directive in a batch job:
    #SBATCH --account=my_project@gpu

    Comments:

    • The type of hours specified (cpu or gpu) must be in agreement with the partition chosen for the job execution (#SBATCH --partition=cpu_... or #SBATCH --partition=gpu_...) and/or the QoS chosen (#SBATCH --qos=qos_cpu-... or #SBATCH --qos=qos_gpu-...).
      For more information, you may consult our documentation detailling the partitions and the QoS.
    • The IDRPROJ environment variable indicates the project to which you are currently attached.
      Therefore, its value (echo $IDRPROJ) can be used to define the my_project character string.
    • To know which Unix groups reference your different projects, you can use the Unix groups command.
      For example, for a login attached to two projects, grp1 and grp2 (the generic group gengrp is specific to each login):
      $ groups
      gengrp grp1 grp2
    • To switch to another project, you should use the command eval $(idrenv -d <other_project>).
      With the preceding example, to switch to the grp2 project:
      # Switch to the grp2 project:
      $ eval $(idrenv -d grp2)
       
      # Verification of the selected project:
      $ echo $IDRPROJ
      grp2

The idracct command

A simple call to the idracct command will indicate the CPU and/or GPU hours allocations for each of your projects as well as the hours consumed by each member.

The following is an example of project 999999 with the hours consumption of 2 users, login1 and login2. In this case, 1000000 CPU hours and 40000 GPU hours are allocated for the period of 01 November 2019 to 31 October 2020 :

$ idracct
Derniere mise a jour le 15-11-2019 08:00:01
################################################################################
   
                           PROJET 999999 SUR JEAN-ZAY
################################################################################
================================================================================
 CPU (du 01-11-2019 au 31-10-2020)                   Allocation :  1000000 h.cpu
================================================================================
 
  Compte   Proprietaire                    Consommation Nb travaux Remboursement
  -------- ------------------------------ ------------- ---------- -------------
  login1   Nom1_Prenom1                        42809.00         10             -
  login2   Nom2_Prenom2                        12014.00         50             -
  -------- ------------------------------ ------------- ---------- -------------
                                   Totaux      54823.00         60          0.00
                              Utilisation         0.05%           
 
================================================================================
 GPU (du 01-11-2019 au 31-10-2020)                     Allocation :  40000 h.gpu
================================================================================
 
  Compte   Proprietaire                    Consommation Nb travaux Remboursement
  -------- ------------------------------ ------------- ---------- -------------
  login1   Nom1_Prenom1                        29078.83        248             -
  login2   Nom2_Prenom2                         2706.31        163             -
  -------- ------------------------------ ------------- ---------- -------------
                                   Totaux      31785.14        411          0.00
                              Utilisation        79.46%           

The idracct command has various options. To consult them, use either -h or --help. Here are the most useful:

  • With the -a or --accounting option you can specify the type of hours you want in the command output: either CPU or GPU. Be careful to use capital letters here in indicating the type of hours!
    $ idracct -a GPU
    Derniere mise a jour le 15-11-2019 08:00:01
    ################################################################################
    
                               PROJET 999999 SUR JEAN-ZAY
    ################################################################################
    ================================================================================
     GPU (du 01-11-2019 au 31-10-2020)                     Allocation :  40000 h.gpu
    ================================================================================
     
      Compte   Proprietaire                    Consommation Nb travaux Remboursement
      -------- ------------------------------ ------------- ---------- -------------
      login1   Nom1_Prenom1                        29078.83        248             -
      login2   Nom2_Prenom2                         2706.31        163             -
      -------- ------------------------------ ------------- ---------- -------------
                                       Totaux      31785.14        411          0.00
                                  Utilisation        79.46%           
  • With the -s or --summary option you can obtain the information in a summary format:
    $ idracct -s
    Derniere mise a jour le 15-11-2019 08:00:01
    Projet Categorie Debut      Fin          Budget Unite Consommation NbTravaux Remboursement Utilise
    ------ --------- ---------- ---------- -------- ----- ------------ --------- ------------- -------
    999999 CPU       01-11-2019 31-10-2020  1000000 h.cpu     54823.00        60          0.00   0.05%
    999999 GPU       01-11-2019 31-10-2020    40000 h.gpu     31785.14       411          0.00  79.46%

Note: “Nb Travaux” in the above examples refers to the number of jobs.