Ada : Executing an MPI job in batch under the Intel environment

Jobs are managed on all the nodes by the LoadLeveler software. Jobs are distributed in the classes mainly in function of the elapsed time, the number of cores and the requested memory. You may consult the structure of the classes on Ada here.

To optimise code performance during executions under an Intel environment, IDRIS automatically sets certain environment variables. Consequently, the Intel I_MPI_PIN_DOMAIN is set to allow binding the MPI processes and the threads (hybrid jobs) on the physical cores of the machine.

Attention : A code compiled under intel/2015.2 or intel/2016.2 CANNOT be executed under the IBM environment (via poe): There is incompatibility between the MPI IBM library currently used and the MPI Intel 5.0.xx and 5.1.xx which are part of these new Intel environments.

Attention : The intel/2013.0 environment cannot be used for a batch execution (LoadLeveler job). In fact, the associated mpirun command is bugged and sends back the following error message:

$ mpirun -np 8 ./a.out
[mpiexec@ada295] HYDT_bscd_ll_launch_procs (./tools/bootstrap/external/ll_launch.c:67): ll does not support user-defined host lists
*** glibc detected *** mpiexec.hydra: munmap_chunk(): invalid pointer: 0x00000000024bb660 ***

To submit an MPI job in batch, you must:

  • Create a submission script. The following is an example registered in the intel_mpi.ll file :
# Arbitrary name of the LoadLeveler job
# @ job_name = Intel_mpi
# Standard job output file
# @ output   = $(job_name).$(jobid)
# Job error output file
# @ error= $(job_name).$(jobid)
# Type of job
# @ job_type = mpich
# Number of MPI processes requested
# @ total_tasks = 8
# Permits passing total_tasks to mpirun via NB_TASKS
# @ environment = NB_TASKS=$(total_tasks)
# Job time hh:mm:ss (1h30mn here)
# @ wall_clock_limit = 1:30:00
# @ queue
# Recommendation: Compile and execute your codes under the same Intel environment.
# If necessary, therefore, use the module command to load the appropriate environment.
# For example, if your code is compiled with Intel/2016.2, uncomment the following line:
# module load intel/2016.2
# To have the command echoes
set -x
# Temporary work directory
# The LOADL_STEP_INITDIR variable is automatically set by 
# LoadLeveler at the directory where the llsubmit command is typed
# Execution of an MPI program.
mpirun -np $NB_TASKS ./a.out
  • Submit this script (only from Ada) via the llsubmit command:
$ llsubmit intel_mpi.ll


  • We recommend that you compile and execute your codes under the same Intel environment: Use the same module load intel/… command at the execution and the compilation.
  • In this example, we suppose that the a.out executable is located in the submission directory which is the directory in which the llsubmit command is entered. (The LOADL_STEP_INITDIR variable is automatically set by LoadLeveler.)
  • For an Intel type execution, you must specify the directive # @ job_type = mpich (instead of # @ job_type = parallel for IBM poe).
  • The number of MPI processes is indicated by the # @ total_tasks =… directive as for an IBM MPI job.
  • The binary execution is effected via the mpirun command with the parameter being the total number of MPI processes: mpirun -np $NB_TASKS ./a.out.
  • The Intel_mpi.numero_job calculation output file is also found in the submission directory. It is created at the beginning of the job execution: Editing or modifying it during the execution can disrupt the process.
  • Memory: By default, the memory is 3.5 GB per reserved core (therefore, per MPI task). This value is the maximum which you can request if your job reserves more than 64 cores (keyword total_tasks). If you reserve 64 or fewer cores, the maximum value which you can request is 7.0 GB per reserved core via the keyword # @ as_limit = 7.0gb.
  • If your job contains relatively long sequential commands (pre- or post-processing, transfers or archiving of large files, …), the use of multi-step jobs may be justified.