Jean Zay: Transitioning between two projects

This page contains useful information for Jean Zay users who wish to transfer their workflow from one project to another. This documentation makes reference to the tools for multi-project account management.

For the transition to be possible, the user login must be attached to the two concerned projects : the initial project (called old for the example) and the new project (called new for the example). The information returned by the idrproj command will contain:

$ idrproj
 
Available projects:
-------------------
  old (123456) [default][active]
  new (654321)

Caution: The two projects must coexist throughout the transition process. Therefore, it is necessary for you to effectuate the administrative procedures for opening a new project and then to attach your login to this project before the initial project reaches its expiry date.
We would also like to draw to your attention that after the expiry date has passed, you no longer have access to the computing hours of the initial project. However, the data contained in the disk spaces associated to the initial project are retained and remain accessible for the six months which follow the expiry date.

To modify your default project, you can use the command idrproj -d <new default project>:

$ idrproj -d new
Password: 
Default project changed to new

The following time you connect to Jean Zay, the information returned by the command idrproj will contain:

$ idrproj
 
Available projects:
-------------------
  old (123456)
  new (654321) [default][active]

After this step, all the default configurations regarding connecting, the environment, job submission and accounting will be redirected to the new project.

During the transitioning period, the disk spaces associated to both projects coexist. To designate these disk spaces unambiguously, we advise you to use the environment variables with the prefix that corresponds to the project (« new » or « old » in the example):

$idrenv
...
export new_ALL_CCFRSCRATCH=/...scratchpath.../new/commun; # the common $SCRATCH of new project
export new_ALL_CCFRSTORE=/...storepath.../new/commun;     # the common $STORE of new project
export new_ALL_CCFRWORK=/...workpath.../new/commun;       # the common $WORK of new project
export new_CCFRSCRATCH=/...scratchpath.../new/user;       # the $SCRATCH of new project
export new_CCFRSTORE=/...storepath.../new/user;           # the $STORE of new project
export new_CCFRWORK=/...workpath.../new/user;             # the $WORK of new project
export old_ALL_CCFRSCRATCH=/...scratchpath.../old/commun; # the common $SCRATCH of old project
export old_ALL_CCFRSTORE=/...storepath.../old/commun;     # the common $STORE of old project
export old_ALL_CCFRWORK=/...workpath.../old/commun;       # the common $WORK of old project
export old_CCFRSCRATCH=/...scratchpath.../old/user;       # the $SCRATCH of old project
export old_CCFRSTORE=/...storepath.../old/user;           # the $STORE of old project
export old_CCFRWORK=/...workpath.../old/user;             # the $WORK of old project
...

The data can be transfered from one space to another in the usual way. Example:

$ cp -rp $old_CCFRWORK/mydata $new_CCFRWORK/. 

In addition, you can launch your computations by using the input data stored in the initial disk spaces and direct the outputs to the new disk spaces.

As long as the initial project has not yet expired, you may request that your computing hours be counted on the allocation of either the initial project or the new project, regardless of the disk spaces in which you are working.

The Slurm option to specify in your Slurm scripts or when calling salloc, sbatch or srun is:

• For a job on the CPU partition:

  --account=new@cpu
  # or
  --account=old@cpu 

• For a job on the GPU partition:

  --account=new@v100
  # or
  --account=old@v100