Testing openMPI environment on SGE

#!/bin/bash # #$ -cwd ##$ -j y #$ -S /bin/bash #$ -M [email protected] #$ -pe orte 8 #$ -o output/hello.out #$ -e output/hello.err # # Use modules to setup the runtime environment # . /etc/profile module load compilers/intel/11.1 module load mpi/openmpi/1.4.2/intel # # Execute the run # mpirun -np $NSLOTS ./hello

option	explanation
-cwd	Run in the current working directory
-j y	Stdout and stderr in same output file
-S /bin/bash	Use the bash shell
-M username@...	Mail notifications to this email address
-pe orte 8	Use parallel environment named "orte" for 8 MPI processes
-o filename	Output file for stdout
-e filename	File for stderr

option

explanation

-cwd

Run in the current working directory

-j y

Stdout and stderr in same output file

-S /bin/bash

Use the bash shell

-M username@...

Mail notifications to this email address

-pe orte 8

Use parallel environment named "orte" for 8 MPI processes

-o filename

Output file for stdout

-e filename

File for stderr

Old News ;-)

Running OpenMPI Executables Through PBS

MPI executables need to be launched with the commmand mpirun. The mpirun command takes care of starting up all of the parallel processes in the MPI job. The following is an example PBS script for running an OpenMPI job across four processors. Note that the proper OpenMPI version needs to be loaded before running the job.

#PBS -l nodes=4:ppn=1
#PBS -l walltime=2:00:00
#PBS -j oe

cd $PBS_O_WORKDIR

module load openmpi/gnu
mpirun ./hellompi

More information on the module command and PBS can be found in their respective User Guides.

Open MPI is a Message Passing Interface (MPI) library project combining technologies and resources from several other projects (FT-MPI, LA-MPI, LAM/MPI, and PACX-MPI). It is used by many TOP500 supercomputers including Roadrunner, which was the world's fastest supercomputer from June 2008 to November 2009, and K computer, the fastest supercomputer since June 2011. The Open MPI Project is an open source MPI-2 implementation that is developed and maintained by a consortium of academic, research, and industry partners.

Project website: http://www.open-mpi.org/

Load SGE module

The following Modules files should be loaded for this package:

For GNU:

module load openmpi/openmpi-gnu

For Intel:

module load openmpi/openmpi-intel

Parallel Environment

Use the openmpi parallel environment in your job script (example for a 4 slot job)

#$ -pe openmpi 4

Submit Script

To enable verbose Grid Engine logging for OpenMPI, add the following the mpirun command in the job script --mca pls_gridengine_verbose 1, for example:

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#
#$ -N j_openmpi_hello
#$ -pe openmpi 4
#$ -l h_rt=00:20:00,s_rt=0:18:00
#$ -j y
#
#$ -M [email protected]
#$ -m eas
#
# Load the appropriate module files
. /etc/profile.d/modules.sh
module load openmpi/openmpi-gnu

#$ -V

mpirun --mca pls_gridengine_verbose 1 -np $NSLOTS hello_world_gnu_openmpi

I've installed openmpi , not in /usr/... but in a /commun/data/packages/openmpi/ , it was compiled with --with-sge.

I've added a new PE in SGE as descibed in http://docs.oracle.com/cd/E19080-01/n1.grid.eng6/817-5677/6ml49n2c0/index.html

# /commun/data/packages/openmpi/bin/ompi_info | grep gridengine
MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.3)

# qconf -sq all.q | grep pe_
pe_list               make orte

Without SGE, the program runs without any problem, using several processors.

/commun/data/packages/openmpi/bin/orterun -np 20 ./a.out args

Now I want to submit my program to SGE

In the Open MPI FAQ, I read:

# Allocate a SGE interactive job with 4 slots
# from a parallel environment (PE) named 'orte'
shell$ qsh -pe orte 4

but my output is:

qsh -pe orte 4
Your job 84550 ("INTERACTIVE") has been submitted
waiting for interactive job to be scheduled ...
Could not start interactive job.

I've also tried the mpirun command embedded in a script:

$ cat ompi.sh
#!/bin/sh
/commun/data/packages/openmpi/bin/mpirun  \
    /path/to/a.out args

but it fails

$ cat ompi.sh.e84552
error: executing task of job 84552 failed: execution daemon on host "node02" didn't accept task
--------------------------------------------------------------------------
A daemon (pid 18327) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
error: executing task of job 84552 failed: execution daemon on host "node01" didn't accept task
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.

How can I fix this?

answer in the openmpi mailing list: http://www.open-mpi.org/community/lists/users/2013/02/21360.php

=== answer ===

May 31 at 19:30

Pierre
1 Answer 1

In my case setting "job_is_first_task FALSE" and "control_slaves TRUE" solved the problem.

# qconf -mp mpi1 

pe_name            mpi1
slots              9
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $fill_up
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE

Softpanorama May the source be with you, but remember the KISS principle ;-)	Home	Switchboard	Unix Administration	Red Hat	TCP/IP Networks	Neoliberalism	Toxic Managers
	(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix

News	Compiling openmpi with SGE support	Recommended Links	Installing Mellanox InfiniBand Driver on RHEL 6.5	MCA Parameters	SGE Parallel Environment
InfiniBand	mpirun command	Running MPI jobs	Running jobs under SGE	VASP Performance optimization	Specifying Ethernet interface
ulimit problem with infiniband in SGE		MPI startup(): RLIMIT_MEMLOCK too small problem with SGE and Infiniband
Linux Troubleshooting	GPFS on Red Hat		Admin Horror Stories	Humor	Etc

Top updates <p>Your browser does not support iframes.</p>
Bulletin	Latest	Past week	Past month	Google Search

Testing openMPI environment on SGE

Submit Script

SGE Options

Submitting

NEWS CONTENTS

Old News ;-)

OpenMPI Usage Instructions -- Penn state

OpenMPI - UABgrid Documentation

Load SGE module

Parallel Environment

Submit Script

openmpi - Submitting Open MPI jobs to SGE - Stack Overflow

Recommended Links

Google matched content

Softpanorama Recommended

SGE queues on HPC are described at:

Etc