Ada : MPMD execution of a code coupling in batch

The MPMD (Multiple Program Multiple Data) execution model is supported on Ada. Different types of executables are both launched and communicate with each other by using MPI. All the MPI processes are included within the same MPI_COMM_WORLD communicator.

On Ada, this execution model is realised by using a text file (called runfile in the examples) which contains, on each line, the name of the executable file which will generate the MPI process. The MPI processes are generated by the poe command and its option -pgmmode. The MPI process rank i will correspond to the executable file placed at line i+1 of runfile.

MPMD coupling of MPI codes

As example, the following is a batch job which carries out the coupling of 3 MPI codes; 8 MPI processes are generated:

  • The MPI process rank 0 is generated from the executable ./a.out.
  • The 4 MPI processes from ranks 1 to 4 are generated from ./b.out.
  • The 3 MPI processes from ranks 5 to 7 are generated from ./c.out.
mpmd_mpi.ll
# @ job_name = mpmd
# @ output   = $(job_name).$(jobid)
# @ error    = $(output)
# @ job_type = parallel
# @ total_tasks = 8
# @ wall_clock_limit = 0:30:00
# @ queue
 
# Building the runfile
cat > runfile << EOF
./a.out
./b.out
./b.out
./b.out
./b.out
./c.out
./c.out
./c.out
EOF
 
# MPMD execution using the runfile
poe -pgmmodel mpmd -cmdfile ./runfile

MPMD coupling of hybrid MPI/OpenMP codes

MPMD coupling of hybrid MPI/OpenMP codes is carried out in the same way as in the previous example by using runfile. It is also necessary to specify the number of OpenMP threads of each MPI process. This can be done on each line of runfile by putting the env command before each executable file which is part of the coupling.

The mpmd_mpi_openmp.ll batch job carries out the coupling of 3 MPI/OpenMP codes; 13 MPI processes are generated:

  • The MPI process rank 0 is generated from the executable ./a.out.
  • The 4 MPI processes from ranks 1 to 4 are generated from ./b.out. Each process generates 2 OpenMP threads.
  • The 8 MPI processes from ranks 5 to 12 are generated from ./c.out. Each process generates 4 OpenMP threads.

The resources reserved with the directives of the LoadLeveler batch manager are the following:

  • #@total_tasks=13 (13 MPI processes)
  • #@parallel_threads=4 (the maximum number of threads per MPI process)

In this way, there are 13*4= 52 cores reserved, or 2 Ada nodes.

mpmd_mpi_openmp.ll
# @ job_name = mpmd
# @ output   = $(job_name).$(jobid)
# @ error    = $(output)
# @ job_type = parallel
# @ total_tasks = 13
# @ parallel_threads = 4
# @ wall_clock_limit = 0:30:00
# @ queue
 
# Building the runfile
echo "env OMP_NUM_THREADS=1 ./a_mixte.out" > runfile
 
for i in {1..4}
do
    echo "env OMP_NUM_THREADS=2 ./b_mixte.out" >> runfile
done
 
for i in {1..8}
do
    echo "env OMP_NUM_THREADS=4 ./c_mixte.out" >> runfile
done
 
# MPMD execution using the runfile
poe -pgmmodel mpmd -cmdfile ./runfile