Jean Zay: Disk quotas and commands for visualizing occupation rates

Quotas guarantee equitable access to disk resources. They prevent one group of users from consuming all the space which thereby hinders other groups from working. At IDRIS, quotas limit both the quantity of disk space and the number of files (inodes). These limits are applied per user for the HOME (one HOME per user even if your login is attached to more than one project) and per project for the WORK and the STORE (as many WORK and STORE spaces as there are projects for the same user).

You can consult the occupation of your disk spaces by using the two commands presented on this page:

• idr_quota_user to view your personal usage as a user
• idr_quota_project to view your whole project and the consumption of each of the members

The idrquota command is no longer available. It was the first quota visualisation command which was deployed on Jean Zay. The current idr_quota_user and idr_quota_project commands are an evolution of this.

When a user or project has exceeded a quota, no warning e-mail is sent. Nevertheless, you are informed by error messages such as « Disk quota exceeded » when you manipulate files in the concerned disk space.

When one of the quotas is attained, you can no longer create files in the concerned disk space. Doing this could disturb jobs being executed at that time if they were launched from this space.

Warning: Editing a file after you have reached your disk quota limit can cause the file size to return to zero, thereby deleting its contents.

When you are blocked or in the process of being blocked:

• Try cleaning out the concerned disk space by deleting the files which have become obsolete.
• Archive the directories which you no longer, or rarely, access.
• Move your files into another space in function of their usages (see the page on disk spaces).
• The project manager or deputy may request a quota increase for the STORE space via the Extranet interface.

Comments:

1. You need to remember to check the common disk spaces, $ALL_CCFRWORK and $ALL_CCFRSTORE.
2. A recurrent cause of exceeding quotas is the use of personal conda environments. Please refer to the Python personal environment page to understand the best practices on Jean Zay.

By default, the idr_quota_user command returns your personal user occupation for all of the disk spaces of your active project. For example, if your active project is abc, you will see an output similar to the following:

$ idr_quota_user
 HOME
INODE:   |██-------------------------------| U: 9419/150000  6.28%                                     
STORAGE: |████████████████████████████████-| U: 2.98 GiB/3.00 GiB  99.31%                              
 
ABC STORE
INODE:   |---------------------------------| U: 1/100000  0.00%           G: 12/100000  0.01%          
STORAGE: |---------------------------------| U: 4.00 KiB/50.00 TiB  0.00% G: 48.00 KiB/50.00 TiB  0.00%
 
ABC WORK
INODE:   |███▒▒▒---------------------------| U: 50000/500000 10.00%       G: 100000/500000  20.00%        
STORAGE: |██████████▒▒▒▒▒▒▒▒▒▒-------------| U: 1.25 TiB/5.00 TiB  25.00% G: 2.5 TiB/5.00 TiB 50.00%
 
The quotas are refreshed daily. All the information is not in real time and may not reflect your real
storage occupation.

In this output example, your personal occupation is represented by the black bar and quantified on the right after the letter U (for User). Your personal occupation is also compared to the global occupation of the project which is represented by the grey bar (in this output example) and quantified after the letter G (for Group).

Note that the colours can be different depending on the parameters and/or type of your terminal.

You can refine the information returned by the idr_quota_user command by adding one or more of the following arguments:

• --project def to display the occupation of a different project than your active project (def in this example)
• --all-projects to display the occupation of all the projects to which you are attached
• --space home work to display the occupation of one or more disk spaces in particular (the HOME and the WORK in this example)

Complete help for the idr_quota_user command is accessible by launching:

$ idr_quota_user -h

By default, the idr_quota_project command returns the disk occupation of each member of your active project for all of the disk spaces associated with the project. For example, if your active project is abc, you will see an output similar to the following:

$ idr_quota_project
PROJECT: abc SPACE: WORK
PROJECT USED INODE: 34373/500000 6.87%
PROJECT USED STORAGE: 1.42 GiB/5.00 TiB 0.03%
┌─────────────────┬─────────────────┬─────────────────┬─────────────────┬──────────────────────┐
│      LOGIN      │     INODE ▽     │     INODE %     │     STORAGE     │      STORAGE %       │
├─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────────────────────┤
│      abc001     │            29852│            5.97%│       698.45 MiB│                 0.01%│
│      abc002     │             4508│            0.90%│       747.03 MiB│                 0.01%│
│      abc003     │                8│            0.00%│         6.19 MiB│                 0.00%│
│      abc004     │                1│            0.00%│           0.00 B│                 0.00%│
│      abc005     │                1│            0.00%│           0.00 B│                 0.00%│
└─────────────────┴─────────────────┴─────────────────┴─────────────────┴──────────────────────┘
PROJECT: abc SPACE: STORE
PROJECT USED INODE: 13/100000 0.01%
PROJECT USED STORAGE: 52.00 KiB/50.00 TiB 0.00%
┌─────────────────┬─────────────────┬─────────────────┬─────────────────┬──────────────────────┐
│      LOGIN      │     INODE ▽     │     INODE %     │     STORAGE     │      STORAGE %       │
├─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────────────────────┤
│      abc001     │                2│            0.00%│         8.00 KiB│                 0.00%│
│      abc002     │                2│            0.00%│         8.00 KiB│                 0.00%│
│      abc003     │                2│            0.00%│         8.00 KiB│                 0.00%│
│      abc004     │                2│            0.00%│         8.00 KiB│                 0.00%│
│      abc005     │                1│            0.00%│         4.00 KiB│                 0.00%│
└─────────────────┴─────────────────┴─────────────────┴─────────────────┴──────────────────────┘
The quotas are refreshed daily. All the information is not in real time and may not reflect your real
storage occupation.

A summary of the global occupation is displayed for each disk space, followed by a table with the occupation details of each member of the project.

You can refine the information returned by the idr_quota_project command by adding one or more of the following arguments:

• --project def to display the occupation of a different project than your active project (def in this example)
• --space work to display the occupation of one (or more) particular disk space(s) (the WORK in this example)
• --order storage to display the values of a given column in decreasing order (the STORAGE column in this example)

Complete help for the idr_quota_project command is accessible by launching:

$ idr_quota_project -h

• The projects to which you are attached correspond to the UNIX groups listed by the idrproj command.
• The quotas are not monitored in real time and may not represent the actual occupation of your disk spaces. The idr_quota_user and idr_quota_project commands are updated each morning.
• To know the volumetry in octets and inodes in real time of a given directory (my_directory), you can execute the commands: du -hd0 my_directory and du -hd0 --inodes my_directory, respectively. Contrary to the “idr_quota” commands, the du commands can have a long execution time in relation to the directory size.
• For the WORK and the STORE, the displayed occupation rates include both the personal space ($WORK or $STORE) and the occupation of the common space ($ALL_CCFRWORK or $ALL_CCFRSTORE).