Compiling openmpi with SGE support

SGE support is not enabled by default. For Open MPI v1.3 and later, you need to explicitly request the SGE support with the "--with-sge" command line switch to the Open MPI configure script.

For example:

./configure --with-sge

After Open MPI is installed, you should see one component named gridengine (see also LaoTsao's Weblog )

ompi_info | grep gridengine
                 MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.3)

It is better write a simple script openMI_compile.sh. Here is simplified example

INTEL_ROOT=/opt/intel/composer_xe_2011_sp1.12.361
APPS_ROOT=/Apps
MKLROOT=${INTEL_ROOT}/mkl
PATH=${INTEL_ROOT}/bin/intel64:$PATH
LM_LICENSE_FILE=${INTEL_ROOT}/../licenses
MANPATH=${INTEL_ROOT}/man/en_US/man:$MANPATH
LD_LIBRARY_PATH=${INTEL_ROOT}/compiler/lib/intel64:${MKLROOT}/lib/intel64a

export INTEL_ROOT APPS_ROOT PATH LM_LICENSE_FILE MANPATH LD_LIBRARY_PATH MKLROOT

./configure --prefix=$APPS_ROOT/openmpi-1.6.4 FC=ifort F77=ifort CC=icc CXX=icpc CFLAGS=-m64 CXXFLAGS=-m64 FFLAGS=-m64 FCFLAGS=-m64 --enable-contrib-no-build=vt --with-sge
make all install

NEWS CONTENTS

• general questions and discussion of Open MPI ()
• OpenMPI compile with Open64 AMD Developer Forums
• Lindqvist -- a blog about Linux and Science. Mostly. 181. Compiling openmpi on debian wheezy
• OpenMPI, LSF, InfiniBand, Mellanox OFED and Intel MPI Benchmark what is going on (pdsh -a reboot)

Old News ;-)

general questions and discussion of Open MPI ()

I have a 16-node Mellanox cluster built with Mellanox ConnectX3 cards.
Recently I have updated the MLNX_OFED to the 2.0.5 version. The reason
of this e-mail to the OpenMPI users list is that I am not able to run
MPI applications using the service levels (SLs) feature of the OpenMPI
driver.

Currently, the nodes have the Red-Hat 6.4 with the kernel
2.6.32-358.el6.x86_64. I have compiled OpenMPI 1.6.4 with:

 ./configure --with-sge --with-openib=/usr --enable-openib-connectx-xrc 
--enable-mpi-thread-multiple --with-threads --with-hwloc 
--enable-heterogeneous --with-fca=/opt/mellanox/fca 
--with-mxm-libdir=/opt/mellanox/mxm/lib --with-mxm=/opt/mellanox/mxm 
--prefix=/home/jescudero/opt/openmpi

I have modified the OpenSM code (which is based on 3.3.15) in order to
include a special routing algorithm based on "ftree". Apparently all is
correct with the OpenSM since it returns the SLs when I execute the
command "saquery --src-to-dst slid:dlid". Anyway, I have also tried to
run the OpenSM with the DFSSSP algorithm.

However, when I try to run MPI applications (i.e. HPCC, OSU or even
alltoall.c -included in the OpenMPI sources-) I experience some errors
if the "btl_openib_path_record_info" is set to "1", otherwise (i.e. if
the btl_openib_path_record_info is not enabled) the application
execution ends correctly. I run the MPI application with the next command:

mpirun -display-allocation -display-map -np 8 -machinefile maquinas.aux
--mca btl openib,self,sm --mca mtl mxm --mca
btl_openib_ib_path_record_service_level 1 --mca btl_openib_cpc_include

OpenMPI compile with Open64 AMD Developer Forums

Compile system is AMD Opteron(TM) Processor 6238 stepping 02 (4x 12 core).

I've built Open64 on the same architecture from the src "x86_open64-4.5.2.1-1.src.tar.bz2", with:

../configure --prefix=/cm/shared/progs/AMD/x86_open64-4.5.2.1

The configure options for OpenMPI are:

./configure --prefix=/cm/shared/progs/Libraries/openmpi-1.6.5 CC=opencc CXX=openCC F77=openf90 FC=openf90 FLAGS=-march=bdver1 CXXFLAGS=-march=bdver1 FFLAGS=-march=bdver1 FCFLAGS=-march=bdver1

I've also tried with

./configure --prefix=/cm/shared/progs/Libraries/openmpi-1.6.5 CC=opencc CXX=openCC F77=openf90 FC=openf90 FLAGS=-march=bdver2 CXXFLAGS=-march=bdver2 FFLAGS=-march=bdver2 FCFLAGS=-march=bdver2

For OpenMPI, 'make all' fails with the error:

make[7]: Entering directory `/mnt/fhgfs/home/chemmod/mb123/Software/openmpi-1.6.4/ompi/contrib/vt/vt/tools/vtfilter'

CXX vtfilter-vt_filter.o

In file included from ./vt_filter.cc:13:

./vt_filter.h:39:12: error: floating constant in preprocessor expression

In file included from ./vt_filter.cc:13:

./vt_filter.h:39:12: error: floating constant in preprocessor expression

make[7]: *** [vtfilter-vt_filter.o] Error 1

Anyone have any thoughts on how to resolve this?

Lindqvist -- a blog about Linux and Science. Mostly. 181. Compiling openmpi on debian wheezy

181. Compiling openmpi on debian wheezy

There's nothing complicated about this compilation. It's not a terribly quick build though, and I'm not yet sure exactly what packages are necessary.

sudo apt-get install build-essential gfortran
wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.tar.bz2
tar xvf openmpi-1.6.tar.bz2
cd openmpi-1.6/

sudo mkdir /opt/openmpi/
sudo chown ${USER} /opt/openmpi/

./configure --prefix=/opt/openmpi/1.6/ --with-sge

make
make install

And you're done.

tree -L2 -d /opt/openmpi

Linking to the libs is done as before, although the path to e.g. libmpi.so is /opt/openmpi/1.6/lib/ and not /opt/openmpi/1.6/lib/openmpi/ like in the regular debian package.

└── 1.6

├── bin

├── etc

├── include

│ ├── openmpi

│ └── vampirtrace

├── lib

│ ├── openmpi

│ └── pkgconfig

└── share

├── man

├── openmpi

└── vampirtrace

You might also want to update the /etc/alternatives/libmpi.so symlink.

This is definitely one of those packages where it's worth doing ./configure --help to see what options are available.

Also, I imagine that on ROCKS there may well be a few packages which will have to be compile first and specified using --with-<> switches.

A sample:

--with-blcr(=DIR) Path to BLCR Installation
--with-blcr-libdir=DIR Search for BLCR libraries in DIR
--with-hwloc(=DIR) Build hwloc support. DIR can take one of three
--with-hwloc-libdir=DIR Search for hwloc libraries in DIR. Should only be
--with-valgrind(=DIR) Directory where the valgrind software is installed
--with-memory-manager=TYPE
--with-libpicl(=DIR) Build libpicl support, optionally adding
--with-libpicl-libdir=DIR
--with-timer=TYPE Build high resolution timer component TYPE
--with-portals=DIR Specify the installation directory of PORTALS
--with-portals-libs=LIBS
Libraries to link with for portals
--with-alps Build ALPS scheduler component (default: no)
--with-lsf(=DIR) Build LSF support
--with-lsf-libdir=DIR Search for LSF libraries in DIR
--with-pmi Build PMI support (default: no)
--with-cray-pmi-ext Include Cray PMI2 extensions (default: no)
--with-slurm Build SLURM scheduler component (default: yes)
--with-tm(=DIR) Build TM (Torque, PBSPro, and compatible) support,
--with-ftb(=DIR) Build FTB (Fault Tolerance Backplane) support,
--with-ftb-libdir=DIR Search for FTB (Fault Tolerance Backplane) libraries
--with-esmtp(=DIR) Build esmtp support, optionally adding DIR/include,
--with-esmtp-libdir=DIR Search for the esmtp libraries in DIR
--with-sge Build SGE or Grid Engine support (default: no)
--with-loadleveler Build LoadLeveler scheduler component (default: yes)
--with-elan(=DIR) Build Elan (QsNet2) support, searching for libraries
--with-elan-libdir=DIR Search for Elan (QsNet2) libraries in DIR
--with-mx(=DIR) Build MX (Myrinet Express) support, optionally
--with-mx-libdir=DIR Search for MX (Myrinet Express) libraries in DIR
--with-openib(=DIR) Build OpenFabrics support, optionally adding
--with-openib-libdir=DIR
--with-portals(=DIR) Build Portals support, optionally adding
--with-portals-config configuration to use for Portals support. One of
--with-portals-libs=LIBS
Libraries to link with for portals
--with-sctp(=DIR) Build SCTP support, searching for libraries in DIR
--with-sctp-libdir=DIR Search for SCTP libraries in DIR
--with-knem(=DIR) Build knem Linux kernel module support, searching
--with-udapl(=DIR) Build uDAPL support, optionally adding DIR/include,
--with-udapl-libdir=DIR Search for uDAPL libraries in DIR
--with-fca(=DIR) Build fca (Mellanox Fabric Collective Accelerator)
--with-io-romio-flags=FLAGS
--with-mxm(=DIR) Build Mellanox Messaging support
--with-mxm-libdir=DIR Search for Mellanox Messaging libraries in DIR
--with-psm(=DIR) Build PSM (Qlogic InfiniPath) support, optionally
--with-psm-libdir=DIR Search for PSM (QLogic InfiniPath PSM) libraries in
--with-contrib-vt-flags=FLAGS
--with-event-rtsig compile with support for real time signals
--with-pic[=PKGS] try to use only PIC/non-PIC objects [default=use
--with-gnu-ld assume the C compiler uses GNU ld [default=no]
--with-sysroot=DIR Search for dependent libraries within DIR

OpenMPI, LSF, InfiniBand, Mellanox OFED and Intel MPI Benchmark what is going on (pdsh -a reboot)

OpenMPI, LSF, InfiniBand, Mellanox OFED and Intel MPI Benchmark: what is going on ?
[email protected] 0600014YT0 | Apr 17 2013 | Tags: mpi hpc-3.2 | 1,565 Visits 0 people like this 0 people like this

I usually use the Intel MPI Benchmark (IMB) as a quick test to check everything is ok from a network point of view (connectivity and performance)

My test cluster for today:

OpenMPI version: 1.6.3
Mellanox OFED version: MLNX_OFED_LINUX-1.5.3-4.0.8 (OFED-1.5.3-4.0.8)
Clustering suite: IBM Platform HPC 3.2 cluster (RHEL 6.2)
Nodes:
1 head node
6 compute nodes (5 package based install + 1 diskless)
Note: the diskless node is not used here

1. MLX OFED install

The OS_OFED components need to be disabled for both the compute (all that are in use, i.e. compute-rhel and compute-diskless-rhel in my case) and installer nodegroups.
You can just mount the ISO image and run the install script. The HCA firmware will be flashed if needed during the process.

2. Open MPI install and configuration
2.1 download OpenMPI and uncompress the tarball

--------------------------------------------------------------------------------

mkdir /shared/compile_temp/openmpi/
cd /shared/compile_temp/openmpi/
wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.3.tar.bz2
tar -xjf openmpi-1.6.3.tar.bz2

--------------------------------------------------------------------------------

2.2 Configure, compile and install OpenMPI
Note: if LSF is in the PATH, then support for it will be automatically enabled. If not sure, you can force it:

--------------------------------------------------------------------------------

cd openmpi-1.6.3
./configure --prefix=/shared/ompi-1.6.3-lsf --enable-orterun-prefix-by-default --with-lsf=/opt/lsf/8.3 --with-lsf-libdir=/opt/lsf/8.3/linux2.6-glibc2.3-x86_64/lib

--------------------------------------------------------------------------------

Note: the path used for the option --with-lsf-libdir is $LSF_LIBDIR

--------------------------------------------------------------------------------

make

make check
make install

--------------------------------------------------------------------------------

2.3 create an environment module for OpenMPI
2.3.1 add the new PATH to MODULEPATH

I added the following to my .bashrc: export MODULEPATH=$MODULEPATH:/home/mehdi/modules

2.3.2 create the module file (/home/mehdi/modules/ompilsf):

--------------------------------------------------------------------------------

#%Module1.0
##
## dot modulefile
##
proc ModulesHelp { } {
global openmpiversion

puts stderr "\tAdds OpenMPI to your environment variables,"
}

module-whatis "adds OpenMPI to your environment variables"

set openmpiversion 1.6.3
set root /shared/ompi-1.6.3-lsf/

prepend-path PATH $root/bin
prepend-path MANPATH $root/man
setenv MPI_HOME $root
setenv MPI_RUN $root/bin/mpirun

prepend-path LD_RUN_PATH $root/lib
prepend-path LD_LIBRARY_PATH $root/lib

--------------------------------------------------------------------------------

2.3.3 check the module works as expected
The first step is to check which modules are currently loaded:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ module list
Currently Loaded Modulefiles:
1) null

--------------------------------------------------------------------------------

Then we can check for the available modules:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ module avail

----------------------------------------------------- /usr/share/Modules/modulefiles -----------------------------------------------------
PMPI/modulefile dot module-cvs module-info modules null use.own

---------------------------------------------------------- /home/mehdi/modules -----------------------------------------------------------
ompi ompilsf

--------------------------------------------------------------------------------

And finally load the module we just created and double check all the environment is properly definedL

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ module load ompilsf
[mehdi@atsplat1 ~]$ module list
Currently Loaded Modulefiles:
1) null 2) ompilsf
[mehdi@atsplat1 ~]$ echo $MPI_HOME
/shared/ompi-1.6.3-lsf/
[mehdi@atsplat1 ~]$ which mpicc
/shared/ompi-1.6.3-lsf/bin/mpicc
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.4 check OpenMPI

2.4.1 verify support for openib and lsf are there:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0, Component v1.6.3)
[mehdi@atsplat1 ~]$ ompi_info | grep lsf
Prefix: /shared/ompi-1.6.3-lsf
MCA ras: lsf (MCA v2.0, API v2.0, Component v1.6.3)
MCA plm: lsf (MCA v2.0, API v2.0, Component v1.6.3)
MCA ess: lsf (MCA v2.0, API v2.0, Component v1.6.3)
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.4.1 copy over an example from the OpenMPI distribution and modify it so the output contains the hostname of the node running the MPI instance:

--------------------------------------------------------------------------------

cp /shared/compile_temp/openmpi/openmpi-1.6.3/examples/hello_c.c .
cp hello_c.c hello_c.orig
vim hello_c.c

--------------------------------------------------------------------------------

The diff is below:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ diff -U4 hello_c.orig hello_c.c
--- hello_c.orig 2013-02-13 13:04:42.074961444 -0600
+++ hello_c.c 2013-02-13 13:06:02.216989733 -0600
@@ -12,13 +12,15 @@

int main(int argc, char* argv[])
{
int rank, size;
+ char hostname[256];

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
- printf("Hello, world, I am %d of %d\n", rank, size);
+ gethostname(hostname,255);
+ printf("Hello, world, I am %d of %d on host %s\n", rank, size, hostname);
MPI_Finalize();

return 0;
}
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.4.2 compile the modified example:

--------------------------------------------------------------------------------

mpicc -o hello hello_c.c

--------------------------------------------------------------------------------

2.4.3 run the example manually, using a host file and outside of LSF

My hostfile is:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat hosts
compute002
compute004
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.4.3.1 run with no specific options and fix any problem

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --hostfile ./hosts ./hello
--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory. This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered. You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

Local host: compute002
Registerable memory: 4096 MiB
Total memory: 32745 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------
Hello, world, I am 0 of 2 on host compute002
Hello, world, I am 1 of 2 on host compute004
[atsplat1:24675] 1 more process has sent help message help-mpi-btl-openib.txt / reg mem limit low
[atsplat1:24675] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

--------------------------------------------------------------------------------

As you can see, the first try is not really great. Let see if everything is fine when using the TCP fabric:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --hostfile ./hosts --mca btl self,tcp ./hello
Hello, world, I am 1 of 2 on host compute004
Hello, world, I am 0 of 2 on host compute002
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

So the problem really comes from my IB setup. Looking at the page mentioned in the output of the job (the first one that used the IB fabric), there are 2 causes for the warning: one related to the Linux kernel and a second one to the locked memory limits.

The max locked memory should not be an issue as my /etc/security/limits.conf on the compute nodes ends with a nice:

* soft memlock unlimited
* hard memlock unlimited

So my issue must be at kernel level

And the following article on developerworks explains very well how to solve the problem:

Looking at the kernel module parameters tells me that I need to change them:

--------------------------------------------------------------------------------

[root@compute004 ~]# cat /sys/module/mlx4_core/parameters/log_num_mtt
0
[root@compute004 ~]# cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
0
[root@compute004 ~]#

--------------------------------------------------------------------------------

And the PAGE_SIZE value is

--------------------------------------------------------------------------------

[root@compute004 ~]# getconf PAGE_SIZE
4096
[root@compute004 ~]#

--------------------------------------------------------------------------------

My nodes have 32G memory so:

max_reg_mem = (2^23) * (2^1) * (4 kB) = 64 GB

which means:

log_num_mtt = 23

log_mtts_per_seg = 1

The way to get the proper values is to add the following line to /etc/modprobe.d/mlx4_en.conf :

--------------------------------------------------------------------------------

options mlx4_core log_num_mtt=23 log_mtts_per_seg=1

--------------------------------------------------------------------------------

Then we need to restart openibd so the modules are unloaded / reloaded and check the new values of log_mtts_per_seg and log_num_mtt:

--------------------------------------------------------------------------------

[root@compute004 ~]# cat /sys/module/mlx4_core/parameters/log_num_mtt
23
[root@compute004 ~]# cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
1

--------------------------------------------------------------------------------

Perfect, now it is time to run the example again:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --hostfile ./hosts ./hello
Hello, world, I am 1 of 2 on host compute004
Hello, world, I am 0 of 2 on host compute002
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

So now we can force the use of the IB fabric only:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --hostfile ./hosts --mca btl self,openib ./hello
Hello, world, I am 1 of 2 on host compute004
Hello, world, I am 0 of 2 on host compute002
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.4.3.2 Final fix in the IBM Platform HPC framework
We use the CFM framework to append the options to the /etc/modprobe.d/mlx4_en.conf file.
In the CFM directory for the compute nodegroup in use. In my case:

/etc/cfm/compute-rhel-6.2-x86_64eth0/

create the appropriate directory and file:

etc/modprobe.d/mlx4_en.conf.append

The mlx4_en.conf.append file contains only 2 lines:

Then run cfmsync -f -n compute-rhel-6.2-x86_64eth0

2.5 compile IMB

--------------------------------------------------------------------------------

mkdir /shared/compile_temp/IMB/
cd /shared/compile_temp/IMB/

--------------------------------------------------------------------------------

Assuming the tarball is already there:

--------------------------------------------------------------------------------

tar -xzf IMB_3.2.3.tgz

--------------------------------------------------------------------------------

Then we need to go in the src directory and create a make file for OpenMPI:

--------------------------------------------------------------------------------

[root@atsplat1 IMB]# cd imb_3.2.3/src/

[root@atsplat1 src]# cp make_mpich make_ompi

--------------------------------------------------------------------------------

The setup of MPI_HOME is commented out, as the module file will set it up.

The diff:

--------------------------------------------------------------------------------

[root@atsplat1 src]# diff -U4 make_mpich make_ompi
--- make_mpich 2011-11-07 16:09:42.000000000 -0600
+++ make_ompi 2013-02-13 09:36:45.616993735 -0600
@@ -1,6 +1,6 @@
# Enter root directory of mpich install
-MPI_HOME=
+#MPI_HOME=

MPICC=$(shell find ${MPI_HOME} -name mpicc -print)

NULL_STRING :=
[root@atsplat1 src]#

--------------------------------------------------------------------------------

Time to compile:

--------------------------------------------------------------------------------

[root@atsplat1 src]# make -f make_ompi

--------------------------------------------------------------------------------

2.6 run IMB to get get the bandwidth associated with each fabric
One of my node showed errors for the InfiniBand card so from now I'll be using a different host file:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat hosts
compute005
compute002
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.6.1 run using native InfiniBand:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --mca btl self,openib --hostfile ./hosts ./IMB-MPI1 pingpong
benchmarks to run pingpong
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V3.2.3, MPI-1 part
#---------------------------------------------------
# Date : Thu Feb 14 08:45:12 2013
# Machine : x86_64
# System : Linux
# Release : 2.6.32-220.el6.x86_64
# Version : #1 SMP Wed Nov 9 08:03:13 EST 2011
# MPI Version : 2.1
# MPI Thread Environment:

# New default behavior from Version 3.2 on:

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

# Calling sequence was:

# ./IMB-MPI1 pingpong

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# PingPong

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.50 0.00
1 1000 1.49 0.64
2 1000 1.26 1.52
4 1000 1.25 3.05
8 1000 1.30 5.89
16 1000 1.30 11.69
32 1000 1.33 22.87
64 1000 1.37 44.60
128 1000 1.98 61.61
256 1000 2.12 115.41
512 1000 2.29 213.54
1024 1000 2.63 370.96
2048 1000 3.46 563.82
4096 1000 4.14 942.99
8192 1000 5.63 1387.80
16384 1000 8.12 1924.25
32768 1000 11.63 2686.07
65536 640 18.67 3347.84
131072 320 32.46 3851.17
262144 160 60.21 4152.16
524288 80 115.53 4327.82
1048576 40 226.39 4417.21
2097152 20 447.33 4471.00
4194304 10 890.95 4489.61

# All processes entering MPI_Finalize

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

The bandwidth and latency are consistents with the hardware capability:

--------------------------------------------------------------------------------

[root@compute004 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:5cf3:fc00:0004:ec2b
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 3: Disabled
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand

Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:5cf3:fc00:0004:ec2c
base lid: 0x10
sm lid: 0x2
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X FDR10)
link_layer: InfiniBand

[root@compute004 ~]#

--------------------------------------------------------------------------------

2.6.2 run using the TCP fabric: surprise !

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --mca btl self,tcp --hostfile ./hosts ./IMB-MPI1 pingpong

...

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 17.15 0.00
1 1000 17.35 0.05
2 1000 17.34 0.11
4 1000 17.36 0.22
8 1000 17.41 0.44
16 1000 17.43 0.88
32 1000 17.74 1.72
64 1000 18.55 3.29
128 1000 19.08 6.40
256 1000 20.74 11.77
512 1000 23.93 20.41
1024 1000 29.15 33.50
2048 1000 269.92 7.24
4096 1000 271.10 14.41
8192 1000 272.24 28.70
16384 1000 273.65 57.10
32768 1000 289.68 107.88
65536 640 790.10 79.10
131072 320 1054.85 118.50
262144 160 1397.37 178.91
524288 80 2552.95 195.85
1048576 40 4664.89 214.37
2097152 20 9089.75 220.03
4194304 10 18022.91 221.94

# All processes entering MPI_Finalize

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

The big surprise is the bandwidth, which is 2 times what is expected for a Gb network. This is explained by how OpenMPI is able to aggregate links when they are in compatible networks. This is explained in more details here. Another noticeable effect of this aggregation is that you are limited by the slowest interface, this is why in this case we are getting 2 times the bandwidth for a Gb interface.

2.6.3 run using only the eth0 interface

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --mca btl self,tcp --mca btl_tcp_if_include eth0 --hostfile ./hosts ./IMB-MPI1 pingpong

...

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 24.52 0.00
1 1000 24.52 0.04
2 1000 24.53 0.08
4 1000 24.53 0.16
8 1000 24.50 0.31
16 1000 24.53 0.62
32 1000 24.53 1.24
64 1000 24.50 2.49
128 1000 24.51 4.98
256 1000 24.55 9.95
512 1000 29.45 16.58
1024 1000 47.34 20.63
2048 1000 551.01 3.54
4096 1000 547.24 7.14
8192 1000 545.31 14.33
16384 1000 551.50 28.33
32768 1000 559.61 55.84
65536 640 1056.22 59.17
131072 320 1391.63 89.82
262144 160 2544.80 98.24
524288 80 4658.02 107.34
1048576 40 9076.14 110.18
2097152 20 18080.40 110.62
4194304 10 35776.65 111.80

# All processes entering MPI_Finalize

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.6.4 run using only the IPoIB interface

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ mpirun -np 2 --mca btl self,tcp --mca btl_tcp_if_include ib1 --hostfile ./hosts ./IMB-MPI1 pingpong
...
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 9.33 0.00
1 1000 9.54 0.10
2 1000 9.54 0.20
4 1000 9.57 0.40
8 1000 9.54 0.80
16 1000 9.53 1.60
32 1000 9.54 3.20
64 1000 9.57 6.38
128 1000 9.72 12.56
256 1000 10.42 23.43
512 1000 10.73 45.50
1024 1000 11.32 86.25
2048 1000 12.62 154.76
4096 1000 14.84 263.19
8192 1000 18.66 418.77
16384 1000 26.97 579.31
32768 1000 47.33 660.20
65536 640 95.72 652.96
131072 320 120.42 1038.04
262144 160 198.02 1262.50
524288 80 342.29 1460.73
1048576 40 656.27 1523.77
2097152 20 1273.80 1570.10
4194304 10 2526.25 1583.38

# All processes entering MPI_Finalize

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.7 Using LSF to submit jobs
2.7.1 basic submission and how to get information about the LSF job
As mentioned earlier, OpenMPI was compiled with support for LSF, which means we can use mpirun natively in bsub scripts / invocations. For example, the following will request 2 job slots (equivalent by default to 2 cores) on 2 different nodes:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ bsub -n 2 -R span[ptile=1] mpirun ./IMB-MPI1 pingpong
Job <3525> is submitted to default queue <medium_priority>.

We can see which nodes are running the job:

[mehdi@atsplat1 ~]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
3525 mehdi RUN medium_pri atsplat1 compute002 * pingpong Feb 14 09:03
compute003

And also get detailed information about the job itself, once done:
[mehdi@atsplat1 ~]$ bhist -l 3525

Job <3525>, User <mehdi>, Project <default>, Command <mpirun ./IMB-MPI1 pingpon
g>
Thu Feb 14 09:03:49: Submitted from host <atsplat1>, to Queue <medium_priority>
, CWD <$HOME>, 2 Processors Requested, Requested Resources
<span[ptile=1]>;
Thu Feb 14 09:03:51: Dispatched to 2 Hosts/Processors <compute002> <compute003>
;
Thu Feb 14 09:03:51: Starting (Pid 28169);
Thu Feb 14 09:03:51: Running with execution home </home/mehdi>, Execution CWD <
/home/mehdi>, Execution Pid <28169>;
Thu Feb 14 09:03:53: Done successfully. The CPU time used is 1.3 seconds;
Thu Feb 14 09:04:01: Post job process done successfully;

Summary of time in seconds spent in various states by Thu Feb 14 09:04:01
PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL
2 0 2 0 0 0 4

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.7.2 how to get the output and verify everything is fine
The best way to double check the information given from LSF is exact is to run our hello job again and get the output written in a file. In our case we'll tag the output and error files with the job number (-o%J.out and -e%J.err):

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ bsub -o%J.out -e%J.err -n 2 -R span[ptile=1] mpirun ./hello
Job <3527> is submitted to default queue <medium_priority>.
[mehdi@atsplat1 ~]$
[mehdi@atsplat1 ~]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
3527 mehdi RUN medium_pri atsplat1 compute003 *n ./hello Feb 14 09:10
compute004
[mehdi@atsplat1 ~]$ ls 3527*
3527.err 3527.out
[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

Good news, the error file is empty and the output file shows a perfect match between the hostname retrieved as part as the MPI job and the nodes allocated by LSF:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat 3527.err
[mehdi@atsplat1 ~]$ cat 3527.out
Sender: LSF System <hpcadmin@compute003>
Subject: Job 3527: <mpirun ./hello> Done

Job <mpirun ./hello> was submitted from host <atsplat1> by user <mehdi> in cluster <atsplat1_cluster1>.
Job was executed on host(s) <1*compute003>, in queue <medium_priority>, as user <mehdi> in cluster <atsplat1_cluster1>.
<1*compute004>
</home/mehdi> was used as the home directory.
</home/mehdi> was used as the working directory.
Started at Thu Feb 14 09:10:20 2013
Results reported at Thu Feb 14 09:10:28 2013

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
mpirun ./hello
------------------------------------------------------------

Successfully completed.

Resource usage summary:

CPU time : 0.19 sec.
Max Memory : 1 MB
Max Swap : 30 MB

Max Processes : 1
Max Threads : 1

The output (if any) follows:

Hello, world, I am 0 of 2 on host compute003
Hello, world, I am 1 of 2 on host compute004

PS:

Read file <3527.err> for stderr output of this job.

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.7.2 Submitting the IMB jobs through LSF

2.7.2.1 using the IB fabric

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ bsub -o%J.out -e%J.err -n 2 -R span[ptile=1] mpirun --mca btl self,openib ./IMB-MPI1 pingpong
Job <3529> is submitted to default queue <medium_priority>.

--------------------------------------------------------------------------------

And the output:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat 3529.out
Sender: LSF System <hpcadmin@compute005>
Subject: Job 3529: <mpirun --mca btl self,openib ./IMB-MPI1 pingpong> Done

Job <mpirun --mca btl self,openib ./IMB-MPI1 pingpong> was submitted from host <atsplat1> by user <mehdi> in cluster <atsplat1_cluster1>.
Job was executed on host(s) <1*compute005>, in queue <medium_priority>, as user <mehdi> in cluster <atsplat1_cluster1>.
<1*compute006>
</home/mehdi> was used as the home directory.
</home/mehdi> was used as the working directory.
Started at Thu Feb 14 09:28:54 2013
Results reported at Thu Feb 14 09:28:56 2013

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
mpirun --mca btl self,openib ./IMB-MPI1 pingpong
------------------------------------------------------------

Successfully completed.

Resource usage summary:

CPU time : 1.25 sec.
Max Memory : 1 MB
Max Swap : 30 MB

Max Processes : 1
Max Threads : 1

The output (if any) follows:

benchmarks to run pingpong
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V3.2.3, MPI-1 part
#---------------------------------------------------
# Date : Thu Feb 14 09:28:55 2013
# Machine : x86_64
# System : Linux
# Release : 2.6.32-220.el6.x86_64
# Version : #1 SMP Wed Nov 9 08:03:13 EST 2011
# MPI Version : 2.1
# MPI Thread Environment:

# New default behavior from Version 3.2 on:

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

# Calling sequence was:

# ./IMB-MPI1 pingpong

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# PingPong

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.42 0.00
1 1000 1.48 0.65
2 1000 1.29 1.48
4 1000 1.25 3.06
8 1000 1.27 5.99
16 1000 1.28 11.90
32 1000 1.30 23.46
64 1000 1.34 45.67
128 1000 1.97 61.82
256 1000 2.11 115.84
512 1000 2.29 213.60
1024 1000 2.64 370.53
2048 1000 3.46 564.81
4096 1000 4.14 943.10
8192 1000 5.65 1383.23
16384 1000 8.13 1922.95
32768 1000 11.67 2678.25
65536 640 18.59 3361.36
131072 320 32.47 3849.49
262144 160 60.17 4155.19
524288 80 115.51 4328.54
1048576 40 226.20 4420.87
2097152 20 447.60 4468.26
4194304 10 890.91 4489.79

# All processes entering MPI_Finalize

PS:

Read file <3529.err> for stderr output of this job.

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.7.2.2 using the TCP fabric

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ bsub -o%J.out -e%J.err -n 2 -R span[ptile=1] mpirun --mca btl self,tcp ./IMB-MPI1 pingpong
Job <3533> is submitted to default queue <medium_priority>.

--------------------------------------------------------------------------------

The output:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat 3533.out
Sender: LSF System <hpcadmin@compute002>
Subject: Job 3533: <mpirun --mca btl self,tcp ./IMB-MPI1 pingpong> Done

Job <mpirun --mca btl self,tcp ./IMB-MPI1 pingpong> was submitted from host <atsplat1> by user <mehdi> in cluster <atsplat1_cluster1>.
Job was executed on host(s) <1*compute002>, in queue <medium_priority>, as user <mehdi> in cluster <atsplat1_cluster1>.
<1*compute005>
</home/mehdi> was used as the home directory.
</home/mehdi> was used as the working directory.
Started at Thu Feb 14 09:37:45 2013
Results reported at Thu Feb 14 09:37:58 2013

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
mpirun --mca btl self,tcp ./IMB-MPI1 pingpong
------------------------------------------------------------

Successfully completed.

Resource usage summary:

CPU time : 25.44 sec.
Max Memory : 1 MB
Max Swap : 30 MB

Max Processes : 1
Max Threads : 1

The output (if any) follows:

benchmarks to run pingpong
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V3.2.3, MPI-1 part
#---------------------------------------------------
# Date : Thu Feb 14 09:37:46 2013
# Machine : x86_64
# System : Linux
# Release : 2.6.32-220.el6.x86_64
# Version : #1 SMP Wed Nov 9 08:03:13 EST 2011
# MPI Version : 2.1
# MPI Thread Environment:

# New default behavior from Version 3.2 on:

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

# Calling sequence was:

# ./IMB-MPI1 pingpong

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# PingPong

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 15.44 0.00
1 1000 15.64 0.06
2 1000 15.63 0.12
4 1000 15.67 0.24
8 1000 15.68 0.49
16 1000 15.75 0.97
32 1000 15.98 1.91
64 1000 16.89 3.61
128 1000 17.40 7.01
256 1000 18.98 12.87
512 1000 22.07 22.12
1024 1000 27.37 35.68
2048 1000 151.39 12.90
4096 1000 183.53 21.28
8192 1000 272.62 28.66
16384 1000 273.90 57.05
32768 1000 284.92 109.68
65536 640 787.86 79.33
131072 320 1054.46 118.54
262144 160 1398.88 178.71
524288 80 2559.74 195.33
1048576 40 4671.30 214.07
2097152 20 9101.65 219.74
4194304 10 17931.90 223.07

# All processes entering MPI_Finalize

PS:

Read file <3533.err> for stderr output of this job.

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.7.2.3 using the interface eth0 only

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ bsub -o%J.out -e%J.err -n 2 -R span[ptile=1] mpirun --mca btl self,tcp --mca btl_tcp_if_include eth0 ./IMB-MPI1 pingpong
Job <3535> is submitted to default queue <medium_priority>.

--------------------------------------------------------------------------------

And the output:

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat 3535.out
Sender: LSF System <hpcadmin@compute002>
Subject: Job 3535: <mpirun --mca btl self,tcp --mca btl_tcp_if_include eth0 ./IMB-MPI1 pingpong> Done

Job <mpirun --mca btl self,tcp --mca btl_tcp_if_include eth0 ./IMB-MPI1 pingpong> was submitted from host <atsplat1> by user <mehdi> in cluster <atsplat1_cluster1>.
Job was executed on host(s) <1*compute002>, in queue <medium_priority>, as user <mehdi> in cluster <atsplat1_cluster1>.
<1*compute005>
</home/mehdi> was used as the home directory.
</home/mehdi> was used as the working directory.
Started at Thu Feb 14 09:40:37 2013
Results reported at Thu Feb 14 09:40:58 2013

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
mpirun --mca btl self,tcp --mca btl_tcp_if_include eth0 ./IMB-MPI1 pingpong
------------------------------------------------------------

Successfully completed.

Resource usage summary:

CPU time : 40.69 sec.
Max Memory : 1 MB
Max Swap : 30 MB

Max Processes : 1
Max Threads : 1

The output (if any) follows:

benchmarks to run pingpong
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V3.2.3, MPI-1 part
#---------------------------------------------------
# Date : Thu Feb 14 09:40:38 2013
# Machine : x86_64
# System : Linux
# Release : 2.6.32-220.el6.x86_64
# Version : #1 SMP Wed Nov 9 08:03:13 EST 2011
# MPI Version : 2.1
# MPI Thread Environment:

# New default behavior from Version 3.2 on:

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

# Calling sequence was:

# ./IMB-MPI1 pingpong

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# PingPong

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 24.55 0.00
1 1000 24.50 0.04
2 1000 24.52 0.08
4 1000 24.52 0.16
8 1000 24.50 0.31
16 1000 24.52 0.62
32 1000 24.57 1.24
64 1000 24.65 2.48
128 1000 24.89 4.90
256 1000 29.11 8.39
512 1000 51.64 9.45
1024 1000 47.34 20.63
2048 1000 218.24 8.95
4096 1000 240.50 16.24
8192 1000 245.51 31.82
16384 1000 324.48 48.15
32768 1000 704.06 44.39
65536 640 1338.30 46.70
131072 320 1702.87 73.41
262144 160 2638.45 94.75
524288 80 4792.39 104.33
1048576 40 9190.72 108.81
2097152 20 18132.53 110.30
4194304 10 35818.20 111.68

# All processes entering MPI_Finalize

PS:

Read file <3535.err> for stderr output of this job.

[mehdi@atsplat1 ~]$

--------------------------------------------------------------------------------

2.7.2.4 using the interface ib1 only

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ bsub -o%J.out -e%J.err -n 2 -R span[ptile=1] mpirun --mca btl self,tcp --mca btl_tcp_if_include ib1 ./IMB-MPI1 pingpong
Job <3534> is submitted to default queue <medium_priority>.

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

[mehdi@atsplat1 ~]$ cat 3534.out
Sender: LSF System <hpcadmin@compute002>
Subject: Job 3534: <mpirun --mca btl self,tcp --mca btl_tcp_if_include ib1 ./IMB-MPI1 pingpong> Done

Job <mpirun --mca btl self,tcp --mca btl_tcp_if_include ib1 ./IMB-MPI1 pingpong> was submitted from host <atsplat1> by user <mehdi> in cluster <atsplat1_cluster1>.
Job was executed on host(s) <1*compute002>, in queue <medium_priority>, as user <mehdi> in cluster <atsplat1_cluster1>.
<1*compute005>
</home/mehdi> was used as the home directory.
</home/mehdi> was used as the working directory.
Started at Thu Feb 14 09:38:26 2013
Results reported at Thu Feb 14 09:38:29 2013

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
mpirun --mca btl self,tcp --mca btl_tcp_if_include ib1 ./IMB-MPI1 pingpong
------------------------------------------------------------

Successfully completed.

Resource usage summary:

CPU time : 4.05 sec.
Max Memory : 1 MB
Max Swap : 30 MB

Max Processes : 1
Max Threads : 1

The output (if any) follows:

benchmarks to run pingpong
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V3.2.3, MPI-1 part
#---------------------------------------------------
# Date : Thu Feb 14 09:38:27 2013
# Machine : x86_64
# System : Linux
# Release : 2.6.32-220.el6.x86_64
# Version : #1 SMP Wed Nov 9 08:03:13 EST 2011
# MPI Version : 2.1
# MPI Thread Environment:

# New default behavior from Version 3.2 on:

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

# Calling sequence was:

# ./IMB-MPI1 pingpong

# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#

# List of Benchmarks to run:

# PingPong

#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 9.20 0.00
1 1000 9.43 0.10
2 1000 9.48 0.20
4 1000 9.47 0.40
8 1000 9.45 0.81
16 1000 9.45 1.62
32 1000 9.46 3.23
64 1000 9.48 6.44
128 1000 9.62 12.69
256 1000 10.35 23.58
512 1000 10.67 45.75
1024 1000 11.37 85.87
2048 1000 12.60 155.04
4096 1000 15.05 259.50
8192 1000 18.58 420.59
16384 1000 26.89 581.07
32768 1000 47.50 657.89
65536 640 95.89 651.77
131072 320 120.43 1037.97
262144 160 197.98 1262.77
524288 80 342.43 1460.15
1048576 40 657.13 1521.78
2097152 20 1275.20 1568.38
4194304 10 2520.35 1587.08

# All processes entering MPI_Finalize

PS:

Read file <3534.err> for stderr output of this job.

[mehdi@atsplat1 ~]$
--------------------------------------------------------------------------------
Modified June 12 2013 by [email protected] 0600014YT0

Recommended Links

FAQ Compiling MPI applications

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: November, 14, 2014