Turing: ABINIT

Introduction

ABINIT is a quantum chemistry software based on the DFT (Density Functional Theory) method.

It was developed by Xavier Gonze and Jean-Michel Beuken of the “Unité de Physico-Chimie et de Physique des Matériaux” of the Catholic University of Leuven, Belgium.

Documentation is available on the ABINIT Website.

Availability

The available versions are:

  • 7.4.1 MPI (default version)
  • 7.4.1 hybrid MPI/OpenMP
  • 7.10.4 hybrid MPI/OpenMP

The parallel MPI and MPI/OpenMP hybrid versions (if appropriate) are installed. Attention : The hybrid versions are still being developed. They give good performance but we advise you to perform some tests to verify your results.

Launching script

The following is an example of a hybrid parallelisation launching script for a calculation run in WORKDIR:

job.ll
# @ job_name         = ABINIT
# @ output           = $(job_name).$(jobid)
# @ error            = $(job_name).$(jobid)
# @ job_type         = BLUEGENE
# @ bg_size          = 64
# @ wall_clock_limit = 1:00:00
# @ queue
 
### Module initialisation ###
module load abinit/7.4.1/hyb
 
### Command echoes ###
set -x
 
### Run the calculation ###
runjob --np 1024 --ranks-per-node=16 --envs "OMP_NUM_THREADS=4" : $ABINIT_EXEDIR/abinit < abinit.files > out.log

The following is an example of an MPI parallelisation launching script for a calculation run in TMPDIR:

job.ll
# @ job_name         = ABINIT
# @ job_type         = BLUEGENE
# @ output           = $(job_name).$(jobid)
# @ error            = $(job_name).$(jobid)
# @ wall_clock_limit = 1:00:00
# @ bg_size          = 64
# @ queue
 
### Module initialisaion ###
module load abinit
 
### Command echoes ###
set -x
 
### Copy to the TMPDIR ###
cp ./* $TMPDIR
 
### Run the calculation ###
cd $TMPDIR
runjob --np 1024 --ranks-per-node=16 : $ABINIT_EXEDIR/abinit < abinit.files > out.log
 
### Copy to the submission folder ###
cd -
cp $TMPDIR/* .

More complete advice about Turing job submission in queue may be found on Execution of a parallel MPI code in batch or Execution of a hybrid MPI/OpenMP parallel code in batch.

Parameters specific to the software

Pseudo-potentials

Two complete sets of pseudopotentials for the whole periodic table are available on the ABINIT Website.

File: abinit.files

In the above example of launching scripts, abinit.files is necessary. It contains the list of input files, the prefixes for the output files, as well as the list of pseudopotentials for calculation. If this file is not present on the executable command line, your calculation will stay blocked (as the code waits for this file list). The file is typically constructed as follows:

input_file.in
output_file.out
file_prefixes_temp
file_prefixes_wave function (etc)
file_prefixes_log_of_each_process
/workgpfs/rech/xxx/rxxx000/ABINIT/pseudo1.psp
/workgpfs/rech/xxx/rxxx000/ABINIT/pseudo2.psp

Parallelisation by default

ABINIT uses MPI parallelisation in numerous parts of its code. Some of these parts could benefit from an OpenMP parallelisation. By default, the parallelisation is done on the k points and/or on the spin states.

For best use of the resources, it is necessary to choose a number of MPI tasks which is a multiple of the number of k points (for example, without spin and for 500 k points, 500 MPI tasks will give a more rapid solution than would 512 tasks). Complementary information can be found at ABINIT tutorials.

Automatic parallelisation (7.10+ versions)

In order to increase performance significantly compared to the default parallelisation, ABINIT can choose automatically the optimal core distribution accross the calculation steps.

To activate this mode, you have to add in the input file: AUTOPARAL=1.
The core distribution that may already be present in your input file has also to be deleted or commented (keywords npkpt, npband, npfft, max_ncpus, mkmem, paral_kgb, etc.).

KGB parallelisation (versions up to 7.6)

For calculations in the fundamental state, it is possible to obtain an even better performance by using parallelisation on the k points (K), the wave vectors (G) and the bands (B). Unfortunately, this parallelisation method is not automatic using old versions and it is necessary to indicate the number of attributed cores in order to process the k points, the bands, and the Fast Fourier Transforms (FFT). ABINIT is capable of determining the optimal configuration for the calculation but not in an automatic way. A summary of the simplest method is presented below. Other methods are available on the designated page of the ABINIT tutorials. Note that this method functions best for a large number of MPI tasks (> 512):

* The first step is to indicate the following in your input file (for example, input_file.in): paral_kgb=-maximum_number_of_MPI_tasks_requested.
For example, for a maximum of 512 MPI tasks :

paral_kgb=-512
  • Launch this calculation on only 1 MPI task: This is the reason why it is advised to launch this calculation on Ada or on your local machine. This calculation is very rapid and consists only of determining the calculation configuration.
  • For our example, the result is found at the end of the out.log file. (This file is indicated in the job script). The best configuration corresponds to the first line in which the value of weight is the closest to 1.
    Here is an example :

     mpi_setup : COMMENT -
     For dataset=   1  a possible choice for less than  512 processors is:
      nproc     npkpt  npspinor    npband     npfft    bandpp    weight
       480       60        1         4         2         1        0.50
       240       60        1         2         2         2        0.25
       240       60        1         4         1         1        1.00
       120       60        1         2         1         2        1.00
  • You simply need to retrieve from the table the optimal parameters for your calculation. (Please note that on Turing, despite what is indicated, this is the number of MPI tasks and not compute cores). In this example, in choosing 240 MPI tasks (therefore, in your job script: --np 240), you need to indicate paral_kgb=1 in your file input_file.in. (This would need to be indicated in any case to activate parallelization). Therefore, with the chosen parameters, you must indicate:

    paral_kgb=1 
      
    npkpt=60
    npband=4
    npfft=1
  • In this example, to correctly use the Turing resources and maximally use the compute cores, it will be necessary to use the hybrid version of the code. For more information, consult Turing process mapping.

This method remains the simplest and most rapid. By consulting the designated page on the ABINIT site, you will find the complete description of the precise determination of the optimal configuration, parameter by parameter. However, the method thus described could be longer and consume more resources to obtain a result which is often nearly the same result as that of the simple method.