Jean Zay: Computing hours accounting and the idracct command

On Jean Zay, computing hours are allocated per project and with accounting differentiation between CPU and GPU hours.

The accounting hours consumed by your jobs are determined on the basis of:

  • The number of reserved physical cores × the elapsed time for a CPU or GPU job.
  • Or/and the number of reserved GPUs × the elapsed time for a GPU job.

Note : for a GPU type job, the number of reserved GPUs taken into account for the job accounting is not necessarily equal to the number of GPUs requested via the Slurm directive #SBATCH --gres=gpu:.... Indeed, an adjustment can be made according to the number of CPUs which is also requested for the job. For example, requesting a single GPU from a node (i.e. 1/4 or 1/8 of the GPUs depending on the type of node) via #SBATCH --gres=gpu:1 and half of the CPUs of the node, leads to the reservation of half of the resources of the node. The job accounting is then calculated on the basis of a reservation of half of the GPUs (so as if you had requested #SBATCH --gres=gpu:2 or #SBATCH --gres=gpu:4 depending on the type of node).

Note that during an execution in exclusive mode (the --exclusive option of the sbatch command or the Slurm directive #SBATCH --exclusive), you reserve all of the compute nodes:

  • 40 physical cores for cpu partitions.
  • 40 physical cores and 4 GPUs for default gpu partition.
  • 24 physical cores and 8 GPUs for gpu_p2 partition.
  • 48 physical cores and 8 GPUs for gpu_p5 partition.

The accounting will then be on the basis of:

  • The number of reserved nodes × 40 cores × elapsed time for a CPU job.
  • The number of reserved nodes × 4 GPUs × elapsed time for a GPU job in default gpu partition..
  • The number of reserved nodes × 8 GPUs × elapsed time for a GPU job in gpu_p2 and gpu_p5 partitions.

IMPORTANT: All executions requiring one or more compute nodes are automatically run in exclusive mode and thus are invoiced on the basis of the number of full nodes reserved.

You can consult your project allocation by using the idracct command (see section below).

Moreover, we recommend that you read the documentation about the Regulating of computing hours which influences the number and usage of computing hours.

Hours accounting

The hours accounting depends on if your login is attached to a single project or to several projects, and if the projects have CPU, V100 GPU, A100 GPU hours or at least two of three accountings:

  • If your single project has only CPU or GPU hours (V100 or A100), you do not need to specify for which allocation the consumed computing hours should be counted.
  • In contrast, if your single project has both CPU and GPU hours or both V100 GPU and A100 GPU hours or if your login is attached to 2 or more projects, you must imperatively specify the hours allocation on which your consumed computing hours should be counted. Without this information, your consumed hours could be counted for a different project than the one on which you think you are working.
    It is obligatory, therefore, that you add either the Slurm directive #SBATCH --account=... in the header of the submission script, or the --account=... option to the sbatch command, taking care to specify the right project (my_project) and the type of hours to use (cpu, v100 or a100):
    • For CPU hours
      # As argument of the sbatch, salloc or srun commands:
      $ sbatch ... --account=my_project@cpu job_cpu.slurm
       
      # Or via the SLURM directive in a batch job:
      #SBATCH --account=my_project@cpu
    • For V100 GPU hours
      # As argument of the sbatch, salloc or srun commands:
      $ sbatch ... --account=my_project@v100 job_gpu.slurm
       
      # Or via the SLURM directive in a batch job:
      #SBATCH --account=my_project@v100
    • For A100 GPU hours
      # As argument of the sbatch, salloc or srun commands:
      $ sbatch ... --account=my_project@a100 job_gpu.slurm
       
      # Or via the SLURM directive in a batch job:
      #SBATCH --account=my_project@a100

Comments:

  • The type of hours specified (cpu, v100 or a100) must be in agreement with the partition chosen for the job execution (#SBATCH --partition=cpu_... or #SBATCH --partition=gpu_...) and/or the QoS chosen (#SBATCH --qos=qos_cpu-... or #SBATCH --qos=qos_gpu-...).
    For more information, you may consult our documentation detailling the partitions and the QoS.
  • The IDRPROJ environment variable indicates the active project to which you are currently attached.
    Therefore, its value (echo $IDRPROJ) can be used to define the my_project character string.
  • To know all possible values to set the my_project character string and which reference your different projects, you can use the IDRIS command idrproj (see idrproj -h). For example, for a login attached to two projects, prj1 and prj2:
    $ idrproj
     
    Available projects:
    -------------------
      prj1 (999999) [default][active]
      prj2 (999998)

    In this case, my_project can be set to prj1 or prj2.

The idracct command

A simple call to the idracct command will indicate the CPU and/or GPU hours allocations for each of your projects as well as the hours consumed by each member.

The following is an example of project 999999 with the hours consumption of 2 users, login1 and login2. In this case, 1000000 CPU hours and 40000 V100 GPU hours are allocated for the period of 01 November 2019 to 31 October 2020 :

$ idracct
Derniere mise a jour le 15-11-2019 08:00:01
################################################################################
   
                           PROJET 999999 SUR JEAN-ZAY
################################################################################
================================================================================
 CPU (du 01-11-2019 au 31-10-2020)                   Allocation :  1000000 h.cpu
================================================================================
 
  Compte   Proprietaire                    Consommation Nb travaux Remboursement
  -------- ------------------------------ ------------- ---------- -------------
  login1   Nom1_Prenom1                        42809.00         10             -
  login2   Nom2_Prenom2                        12014.00         50             -
  -------- ------------------------------ ------------- ---------- -------------
                                   Totaux      54823.00         60          0.00
                              Utilisation         0.05%           
 
================================================================================
 V100 (du 01-11-2019 au 31-10-2020)                    Allocation :  40000 h.gpu
================================================================================
 
  Compte   Proprietaire                    Consommation Nb travaux Remboursement
  -------- ------------------------------ ------------- ---------- -------------
  login1   Nom1_Prenom1                        29078.83        248             -
  login2   Nom2_Prenom2                         2706.31        163             -
  -------- ------------------------------ ------------- ---------- -------------
                                   Totaux      31785.14        411          0.00
                              Utilisation        79.46%           

The idracct command has various options. To consult them, use either -h or --help. Here are the most useful:

  • With the -a or --accounting option you can specify the type of hours you want in the command output: either CPU or V100 or A100. Be careful to use capital letters here in indicating the type of hours!
    $ idracct -a V100
    Derniere mise a jour le 15-11-2019 08:00:01
    ################################################################################
    
                               PROJET 999999 SUR JEAN-ZAY
    ################################################################################
    ================================================================================
     V100 (du 01-11-2019 au 31-10-2020)                    Allocation :  40000 h.gpu
    ================================================================================
     
      Compte   Proprietaire                    Consommation Nb travaux Remboursement
      -------- ------------------------------ ------------- ---------- -------------
      login1   Nom1_Prenom1                        29078.83        248             -
      login2   Nom2_Prenom2                         2706.31        163             -
      -------- ------------------------------ ------------- ---------- -------------
                                       Totaux      31785.14        411          0.00
                                  Utilisation        79.46%           
  • With the -s or --summary option you can obtain the information in a summary format:
    $ idracct -s
    Derniere mise a jour le 15-11-2019 08:00:01
    Projet Categorie Debut      Fin          Budget Unite Consommation NbTravaux Remboursement Utilise
    ------ --------- ---------- ---------- -------- ----- ------------ --------- ------------- -------
    999999 CPU       01-11-2019 31-10-2020  1000000 h.cpu     54823.00        60          0.00   0.05%
    999999 V100      01-11-2019 31-10-2020    40000 h.gpu     31785.14       411          0.00  79.46%

Note: “Nb Travaux” in the above examples refers to the number of jobs.