SGE Parallel Environment

• Introduction
• Round Robin vs Fill Up Modes
• Important details
• MPI jobs and SGE parallel environment
• Allocation rule
• Typical operations
• Useful commands

Introduction

Parallel environment (PE) is the central notion of SGE and represents  a set of settings that tell Grid Engine how to start, stop, and manage jobs run by the class of  queues that is using this PE. It also set some parameters for parallel messaging framework such as MPI, that is used by parallel jobs.

The usual syntax applies:

• Show me the list of all PE created       qconf -spl
• Show me partucular PE               qconf -sp <PE name>
• Let me change it                    qconf -mp <PE name>
• Create new from file                 qconf -Ap ./my-PE-template.txt

Parallel environment is the defining characteristic of each queue. Needs to be specified correctly for queue to work. It is present in each queue definition as  pe_list attribute which can contain a single PE or list of PEs.  For example:

pe_list               make mpi mpi_fill_up

Each parallel environment determines a class of queues that use it and has several important attributes are:

1. slots - the maximum number of job slots that the parallel environment is allowed to occupy at once
2. allocation_rule" -> see man page. $pe_slots will allocate all slots for that job on a single host. Other rules support to schedule the job on multiple machines.
3. control_slaves -> when set to "true" Grid Engine takes care about  starting the slave MPI taks. In this case MPI should be compiled with the option -with_sge
4. job_is_first_task  The job_is_first_task parameter can be set to TRUE or FALSE. A value of TRUE indicates that the Sun Grid Engine job script already contains one of the tasks of the parallel application (the number of slots reserved for the job is the number of slots requested with the -pe switch), while a value of FALSE indicates that the job script (and its child processes) is not part of the parallel program (the number of slots reserved for the job is the number of slots requested with the -pe switch + 1).

   If wallclock accounting is used (execd_params ACCT_RESERVED_USAGE and/or SHARETREE_RESERVED_USAGE set to TRUE) and control_slaves is set to FALSE, the job_is_first_task parameter influences the accounting for the job: A value of TRUE means that accounting for cpu and requested memory gets multiplied by the number of slots requested with the -pe switch, if job_is_first_task is set to FALSE, the accounting information gets multiplied by number of slots + 1.
    

5. accounting_summary This parameter is only checked if control_slaves (see above) is set to TRUE and thus Sun Grid Engine is the creator of the slave tasks of a parallel application via sge_execd(8) and sge_shepherd(8). In this case, accounting information is available for every single slave task started by Sun Grid Engine.

   The accounting_summary parameter can be set to TRUE or FALSE. A value of TRUE indicates that only a single accounting record is written to the accounting(5) file, containing the accounting summary of the whole job including all slave tasks, while a value of FALSE indicates an individual accounting(5) record is written for every slave task, as well as for the master task.
   
   Note: When running tightly integrated jobs with SHARETREE_RESERVED_USAGE set, and with having accounting_summary enabled in the parallel environment, reserved usage will only be reported by the master task of the parallel job. No per parallel task usage records will be sent from execd to qmaster, which can significantly reduce load on qmaster when running large tightly integrated parallel jobs.

Round Robin vs Fill Up Modes

Notice the allocation_rule setting in the output of the qconf command in the previous section. This defines how to assign slots to a job. By default StarCluster configures round_robin allocation. This means that if a job requests 8 slots for example, it will go to the first machine, grab a single slot if available, move to the next machine and grab a single slot if available, and so on wrapping around the cluster again if necessary to allocate 8 slots to the job.

You can also configure the parallel environment to try and localize slots as much as possible using the fill_up allocation rule. With this rule, if a user requests 8 slots and a single machine has 8 slots available, that job will run entirely on one machine. If 5 slots are available on one host and 3 on another, it will take all 5 on that host, and all 3 on the other host. In other words, this rule will greedily take all slots on a given node until the slot requirement for the job is met.

You can switch between round_robin and fill_up modes using the following command:

$ qconf -mp mpi 

Where mpi is the name of pa PE. This will open up vi (or any editor defined in EDITOR env variable) and let you edit the parallel environment settings. To change from round_robin to fill_up in the above example, change the allocation_rule line from:

allocation_rule    $round_robin

to:

allocation_rule    $fill_up

After making the change and saving the file you can verify your settings using:

sgeadmin@ip-10-194-13-219:~$ qconf -sp mpi
pe_name            mpi
slots              16
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $fill_up
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE

Important details

Some important details are well explained in the blog post Configuring a New Parallel Environment

 Mar 23, 2007 | DanT's Grid Blog

By templedf on  Mar 23, 2007

Since this seems to be a regular topic on the user mailing list, here's a quick guide to setting up a parallel environment on Grid Engine:

1. First, create/borrow/steal the startup and shutdown scripts for the parallel environment you're using. You can find MPI and PVM scripts in the $SGE_ROOT/mpi and $SGE_ROOT/pvm directories, respectively. If you cannot find scripts for your parallel environment, you'll have to create them. The startup script must prepare the parallel environment for being used. With most MPI implementations, that's just a matter of creating a "machines" file that lists the machines which are to run the parallel job. The shutdown script must clean up after the parallel job's execution. The MPI shutdown script can just delete or rename the "machines" file.
    
2. Next, you have to tell Grid Engine about your parallel environment. You can do that interactively with qmon or qconf -ap <pe_name> or you can write the data to a file and use qconf -Ap <file_name>.

   For an example of what such a file would look like, see $SGE_ROOT/mpi/mpi.template or $SGE_ROOT/pvm/pvm.template.

   Let's look at what the parallel environment configuration contains.

   pe_name           template
   slots             999
   user_lists        NONE
   xuser_lists       NONE
   start_proc_args   /bin/true
   stop_proc_args    /bin/true
   allocation_rule   $pe_slots
   control_slaves    FALSE
   job_is_first_task FALSE
   urgency_slots     min

   • pe_name - the name by which the parallel environment will be known to Grid Engine
   • slots - the maximum number of job slots that the parallel environment is allowed to occupy at once
   • users_lists - an ACL specifying the users who are allowed to use the parallel environment. If set to NONE, it means any user can use it
   • xusers_list - an ACL specifying the users who are not allowed to use the parallel environment. Users in both the users_list and xusers_list are not allowed to use the parallel environment
   • start_proc_args - the path to the startup script for the parallel environment followed by any needed arguments. Grid Engine provides some inline variables that you can use as arguments:
     • $pe_hostfile - the path to a file written by Grid Engine which contains information about how and where the parallel job should be run
     • $host - the host on which the parallel environment is being started
     • $job_owner - the name of the user who owns the parallel job
     • $job_id - the id of the parallel job
     • $job_name - the name of the parallel job
     • $pe - the name of the parallel environment
     • $pe_slots - the number of job slots assigned to the job
     • $queue - the name of the queue in which the parallel job is running

     The value of this setting is the command that will be run to start the parallel environment for every parallel job.

   • stop_proc_args - the path to the shutdown script for the parallel environment followed by any needed arguments. The same inline variables are available as with start_proc_args.
   • allocation_rule - this setting controls how job slots are assigned to hosts. It can have four possible values:
     • a number - if set to a number, Grid Engine will assign that many slots to the parallel job on each host until the assigned number of job slots is met. Setting this attribute to 1, for example, would mean that the job gets a single job slot on each host where it is assigned. Grid Engine will not assign the job more job slots than the number of assigned hosts multiplied by this attribute's value.
     • $fill_up - use all of the job slots on a given host before moving to the next host
     • $round_robin - select one slot from each host in a round-robin fashion until all job slots are assigned. This setting can result in more than one job slot per host.
     • $pe_slots - place all the job slots on a single machine. Grid Engine will only schedule such a job to a machine that can host the maximum number of slots requested by the job. (See below.)
   • control_slaves - this setting tells Grid Engine whether the parallel environment integration is "tight" or "loose". See your parallel environment's documentation for more details.
   • job_is_first_task - this setting tells Grid Engine whether the first task of the parallel job is actually a job task or whether it's just there to kick off the rest of the jobs. This setting is also determined by your parallel environment integration.
   • urgency_slots - this setting affect how resource requests affect job priority for parallel jobs. The values can be "min," "max," "avg," or a number. For more information about resource-based job priorities, see this white paper

   For more information about these settings, see the sge_pe man page.
    

3. The next step is to enable your parallel environment for the queues where it should be available. You can add the parallel environment to a queue interactively with qmon or qconf -mq <queue> or in a single action with qconf -aattr queue pe_list <pe_name> <queue>.
    
4. Now you're ready to test your parallel environment. Run qsub -pe <pe_name> <slots>. Aside from the usual output and error files (<job_name>.o<job_id> and <job_name>.e<job_id>, respectively), you should also look for the parallel environment startup output and error files, <job_name>.po<job_id> and <job_name>.pe<job_id>.

That's all there is to it! Just to make sure we're clear on everything, let's do an example. Let's create a parallel environment that starts up an RMI registry and stores the port number in a file so that the job can find it.

First thing we have to do is write the startup and shudown scripts for the RMI parallel environment. Here's what they look like:

rmi_startup.sh

#!/bin/sh
# $TMPDIR and $JOB_ID are set by Grid Engine automatically

# Borrowed from $SGE_ROOT/mpi/startmpi.sh
PeHostfile2MachineFile()
{
   cat $1 | while read line; do
      host=`echo $line|cut -f1 -d" "|cut -f1 -d"."`
      nslots=`echo $line|cut -f2 -d" "`
      i=1

      while [ $i -le $nslots ]; do
         echo $host
         i=`expr $i + 1`
      done
   done
}

# get arguments
pe_hostfile=$1

# ensure pe_hostfile is readable
if [ ! -r $pe_hostfile ]; then
   echo "$me: can't read $pe_hostfile" >&2
   exit 1
fi

# create machines file
machines="$TMPDIR/machines"
PeHostfile2MachineFile $pe_hostfile >> $machines

# We use ports 40000-40999
port=`expr \\( $JOB_ID % 1000 \\) + 40000`

# Start the registry
/usr/java/bin/rmiregistry $port &

# Save the registry's PID so that we can stop it later
echo $! > $TMPDIR/pid

# Save the port number so the job can find it
echo $port > $TMPDIR/port

rmi_shutdown.sh

#!/bin/sh
# $TMPDIR is set by Grid Engine automatically

# Get the registry's PID
pid=`cat $TMPDIR/pid`

# Kill the registry
kill $pid

# Clean up the files the startup script created
rm $TMPDIR/pid
rm $TMPDIR/port
rm $TMPDIR/machines

Next thing we have to do is add our parallel environment to Grid Engine. First we create a file, say /tmp/rmi_pe, with the following contents:

pe_name           rmi
slots             4
user_lists        NONE
xuser_lists       NONE
start_proc_args   /home/dant/rmi_startup.sh $pe_hostfile
stop_proc_args    /home/dant/rmi_shutdown.sh
allocation_rule   $round_robin
control_slaves    TRUE
job_is_first_task FALSE
urgency_slots     min

Note that control_slaves is true and job_is_first_task is false. Because we're writing the integration scripts, the choice is somewhat arbitrary, but it affects how the job scripts must be written, as we'll see below. It also affect whether the qmaster is able to keep accounting records on the slave tasks. If control_slaves is false, the qmaster is have no records of how much resources the slaves tasks consumed.

Now we add the parallel environment with qconf -Ap /tmp/rmi_pe. We could have skipped a step by running qconf -ap rmi and entering the data in the editor that comes up, but they way we've done it here is scriptable.

The next step is to add our parallel environment to our queue with qconf -aattr queue pe_list rmi all.q. Again, we could have run qconf -mq all.q and edited the pe_list attribute in the editor, but the way we've done it is scriptable.

Last thing to do is test out our parallel environment. First we need a job script:

#!/bin/sh
#$ -S /bin/sh

port=`cat $TMPDIR/port`
qrsh=$SGE_ROOT/bin/$ARC/qrsh

cat $TMPDIR/machines | while read host; do
   $qrsh -inherit $host /usr/bin/java -cp ~/rmi.jar RMIApp $port &
done

Let's look at this job script for a moment. The first thing to notice is the use of qrsh -inherit. The -inherit switch is specifically for kicking off slave tasks. It requires that the target host name be supplied. In order to get the target host name, we read the machines file that the startup script generated from the one Grid Engine supplied.

The second thing to notice is how ugly the use of qrsh -inherit is. RMI is not really a parallel environment. It's a communications framework. It doesn't do the work of kicking off remote processes for you. So, instead, we have to do it ourselves in the job script. With a true parallel environment, like any of the MPI flavors, the framework also takes care of starting the remote processes, often through rsh. In the MPI scripts included with Grid Engine, an rsh wrapper script is included, which transparently replaces calls to rsh with calls to qrsh -inherit. By using that wrapper script, the parallel environment's calls to rsh can be rerouted through the grid via qrsh without having to modify the parallel environment itself to work with Grid Engine.

The last thing to notice is how this script correlates to the control_slaves and job_is_first_task attributes of the parallel environment configuration. Let's start with first_job_is_task. In our configuration, we set it to false. That means that this master job script is not counted as a job and does no real work. That is why our script doesn't do anything but kick off sub-tasks. If job_is_first_task had been true, our job script would be expected to run one of the RMIApp instances itself.

Now let's talk about the control_slaves attribute. If control_slaves is true, we are allowed to use qrsh -inherit to kick off our sub-tasks. The qmaster will not, however, allow us to kick off more subtasks than the number of slots we've been assigned (minus 1 if job_is_first_task is true). The advantage of using qrsh -inherit is that the sub-tasks are tracked by Grid Engine like regular jobs. If control_slaves is false, we have to use some mechanism external to Grid Engine, such as rsh or ssh, to kick off our sub-tasks, meaning that Grid Engine cannot track them and is actually fully unaware of them. That's why job_is_first_task is meaningless when control_slaves is false.

In order to test our job we need a Java application called RMIApp. As that's outside the scope of the example, let's just pretend we have a parallel Java application that uses the RMI registry for intra-process communication. To submit our job we use qsub -pe rmi 2-4 rmi_job.sh. The -pe rmi 2-4 argument tells the qmaster that we're using the rmi parallel environment and we want 4 job slots assigned to our job, but we will accept as few as 2. Because our job script starts a sub-task for every entry in to host file, it will start the right number of sub-tasks, no matter how many slots we are assigned. Had we written the job script to start exactly two sub-tasks, we would have to use -pe rmi 2 so that we could be sure we got exactly two job slots.

While the job is running, run qstat -f. You'll see output something like this:

% qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@ultra20                  BIP   4/10      0.08     sol-amd64
    253 0.55500 rmi_job.sh dant         r     03/23/2007 11:46:51     4

From this output we can see that the job has been scheduled and has been assigned four job slots. Those four job slots only account for the four sub-tasks. The master job itself is not counted because the job_is_first_task attribute is false.

After our job completes, if we look in our home directory (which is where Grid Engine will put the output files since we didn't tell it otherwise), we will find four new files: rmi_job.sh.e253, rmi_job.sh.o253, rmi_job.sh.pe253, and rmi_job.sh.o253, assuming, of course, that our job was number 253. The \*.o253 and \*.e253 files should be familiar. They're the output and error streams from the job script. The \*.po253 and \*.pe253 files are new. They're the output and error streams from the parallel environment startup and shutdown scripts.

So, there you have it. A complete, top-to-bottom example of creating, configuring, and running a parallel environment.

MPI jobs and SGE parallel environment

For most parallel jobs, including those using OpenMP and IntelMPI an SGE Parallel Environment needs to be correctly specified.

This PE acts as glue ensuring that SGE and the parallel, i.e., multi-process, program play nicely together. Two parameters are especially important:

• control_slaves  This parameter can be set to TRUE or FALSE (the default). It indicates whether Sun Grid Engine is the creator of the slave tasks of a parallel application via sge_execd(8) and sge_shepherd(8) and thus has full control over all processes in a parallel application, which enables capabilities such as resource limitation and correct accounting. 

  To gain control over the slave tasks of a parallel application it should be set to TRUE. Example of  PE environments are available through your Sun Grid Engine support. In this case you need to provide access to the file hostfile (which contains the list of nodes on which task should be run in parallel) on all nodes (for example by putting it on NFS).

  Please set the control_slaves parameter to false for all other PE interfaces. For example

  qconf -sp mpi

  pe_name mpi
  slots 999
  user_lists NONE
  xuser_lists NONE
  start_proc_args NONE
  stop_proc_args NONE
  allocation_rule $fill_up
  control_slaves TRUE
  job_is_first_task FALSE
  urgency_slots min
  accounting_summary TRUE

See

man pe_conf

Allocation rule

1. integer_number: An integer number fixing the number of processes per host. If the number is 1, all processes have to reside on different hosts.
2. $pe_slots If the special denominator  $pe_slots is used, the full range of processes as specified with the qsub(1) -pe switch has to be allocated on a single host.
3. $fill_up: Starting from the best suitable host/queue, all available slots are allocated. Further hosts and queues are "filled up" as long as a job still requires slots for parallel tasks.
4. $round_robin: The allocation scheme walks through suitable hosts in a best-suitable-first order.

Typical operations

Show the available parallel environments

qconf -spl

Shown parallel environment

qconf -sp mpi
pe_name           mpi
slots             96
user_lists        NONE
xuser_lists       NONE
start_proc_args   /opt/sge/n1ge6/mpi/startmpi.sh $pe_hostfile
stop_proc_args    /opt/sge/n1ge6/mpi/stopmpi.sh
allocation_rule   $round_robin
control_slaves    FALSE
job_is_first_task TRUE
urgency_slots     min

# qconf -sp ms
pe_name            ms
slots              256
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $pe_slots
control_slaves     FALSE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary FALSE

# qconf -spl
make
ms
 

# qconf -sp make
pe_name make
slots 999
user_lists NONE
xuser_lists NONE
start_proc_args NONE
stop_proc_args NONE
allocation_rule $round_robin
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE 

Add a new parallel environment:

Interactively

qconf -ap mpi180

From the file:

qconf -Ap /home/sgeadmin/config/mpi_fill_up.pe
root@merlin00 added "mpi_fill_up" to parallel environment list

qconf -spl
mpi
mpi_fill_up

From a template

# qconf -sp mpi
pe_name            mpi
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    NONE
stop_proc_args     NONE
allocation_rule    $fill_up
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary TRUE
qsort_args         NONE

qconf -sp mpi_fill_up
pe_name           mpi_fill_up
slots             9999
user_lists        NONE
xuser_lists       NONE
start_proc_args   /opt/sge/mpi/startmpi.sh $pe_hostfile
stop_proc_args    /opt/sge/mpi/stopmpi.sh
allocation_rule   $fill_up
control_slaves    FALSE
job_is_first_task TRUE
urgency_slots     min

Useful commands

Also see the ‘Managing Special Environment’ section in the Administration Guide from sun.com if you need more details about PE configuration.

1. Add the new PE from the file config_file:

   qconf -Ap <config_file>’

2. qconf -spl – view all PEs currently available;
   
3. qconf -sp <PE_name> - list particular PE;
   
4. qconf -dp <PE_name> - remove a PE;
   
5. qconf -mp <PE_name> - modify an existing PE.
6. qconf -sql – to see all queues available;
7. qconf -mq <queue_name> - to modify the queue’s settings.
8. qconf -sq <queue_name> - list queue.

NEWS CONTENTS

• 210517 : How to configure Grid Engine parallel environments in 4 fast steps ( How to configure Grid Engine parallel environments in 4 fast steps, )
• 210517 : openmpi - Submitting Open MPI jobs to SGE ( Stack Overflow )
• 210517 : mpi - Whats the relationship between Sun Grid Engine (SGE) process number and OpenMPI process number ( Jul 29, 2010 , Stack Overflow )
• 210517 : openmpi - Submitting Open MPI jobs to SGE - Stack Overflow ( openmpi - Submitting Open MPI jobs to SGE - Stack Overflow, )
• 210517 : Integrating Intel MPI Library with Sun Grid Engine ( 12/21/2009 )
• 210517 : FAQ Running jobs under SGE ( OpenMPI FAQ )
• 210517 : Sun Grid Engine (SGE) QuickStart ( softpanorama.org, )
• 210517 : Configuring a New Parallel Environment ( Mar 23, 2007 , DanT's Grid Blog )
• 210517 : clustering.gridengine.users - Re pe_slots issues - msg#00048 ( OSDir.com )

Old News ;-)

How to configure Grid Engine parallel environments in 4 fast steps

November 02, 2012 |

Bright Cluster Manager makes so many tasks fast and easy to complete. This article shows how to list, show, and modify Grid Engine parallel environments.

Parallel environments are parallel programming and runtime environments that facilitate the execution of shared memory or distributed memory parallel applications.

Let's get started.

1. First, list existing parallel environments.

# qconf -spl
mpich
mpich2_mpd
mpich2_smpd
mpich_gm
mpich_mx
mvapich
openmpi
openmpi_ib

2. Then, list details for a given parallel environment.

# qconf -sp openmpi_ib
pe_name openmpi_ib
slots 99999
user_lists NONE
xuser_lists NONE
start_proc_args /cm/shared/apps/sge/current/mpi/openmpi/startopenmpi.sh \
-catch_rsh $pe_hostfile
stop_proc_args /cm/shared/apps/sge/current/mpi/openmpi/stopopenmpi.sh
allocation_rule $fill_up
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE

3. Next, modify the existing parallel environment.

# qconf -mp openmpi_ib
pe_name openmpi_ib
slots 128
user_lists NONE
xuser_lists NONE
start_proc_args /cm/shared/apps/sge/current/mpi/openmpi/startopenmpi.sh \
-catch_rsh $pe_hostfile
stop_proc_args /cm/shared/apps/sge/current/mpi/openmpi/stopopenmpi.sh
allocation_rule $fill_up
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE

4. Last, list it to view the changes.

# qconf -sp openmpi_ib
pe_name openmpi_ib
slots 128
user_lists NONE
xuser_lists NONE
start_proc_args /cm/shared/apps/sge/current/mpi/openmpi/startopenmpi.sh \
-catch_rsh $pe_hostfile
stop_proc_args /cm/shared/apps/sge/current/mpi/openmpi/stopopenmpi.sh
allocation_rule $fill_up
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
accounting_summary TRUE

You're done.

openmpi - Submitting Open MPI jobs to SGE

Stack Overflow

I've installed openmpi , not in /usr/... but in a /commun/data/packages/openmpi/ , it was compiled with --with-sge.

I've added a new PE in SGE as descibed in http://docs.oracle.com/cd/E19080-01/n1.grid.eng6/817-5677/6ml49n2c0/index.html

# /commun/data/packages/openmpi/bin/ompi_info | grep gridengine
MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.3)

# qconf -sq all.q | grep pe_
pe_list               make orte

Without SGE, the program runs without any problem, using several processors.

/commun/data/packages/openmpi/bin/orterun -np 20 ./a.out args

Now I want to submit my program to SGE

In the Open MPI FAQ, I read:

# Allocate a SGE interactive job with 4 slots
# from a parallel environment (PE) named 'orte'
shell$ qsh -pe orte 4

but my output is:

qsh -pe orte 4
Your job 84550 ("INTERACTIVE") has been submitted
waiting for interactive job to be scheduled ...
Could not start interactive job.

I've also tried the mpirun command embedded in a script:

$ cat ompi.sh
#!/bin/sh
/commun/data/packages/openmpi/bin/mpirun  \
    /path/to/a.out args

but it fails

$ cat ompi.sh.e84552
error: executing task of job 84552 failed: execution daemon on host "node02" didn't accept task
--------------------------------------------------------------------------
A daemon (pid 18327) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
error: executing task of job 84552 failed: execution daemon on host "node01" didn't accept task
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.

How can I fix this?

---

answer in the openmpi mailing list: http://www.open-mpi.org/community/lists/users/2013/02/21360.php

mpi openmpi sungridengine

=== answer ===

May 31 at 19:30

Pierre
1 Answer 1

In my case setting "job_is_first_task FALSE" and "control_slaves TRUE" solved the problem.

# qconf -mp mpi1 

pe_name            mpi1
slots              9
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $fill_up
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE

mpi - What's the relationship between Sun Grid Engine (SGE) process number and OpenMPI process number

Jul 29, 2010 | Stack Overflow

When running MPI applications in an SGE cluster, I have to assign two parameters of process numbers, one is for SGE itself, and the other is for OpenMPI. For example:

qrsh -pe <pe_name> <number1> mpirun -np <number2> ./program

What are the meanings of number1 and number2 in the command? What's the relationship between them?

If I need 128 (for number2) processes for my MPI application, and I assign 16 to number1, what would happen?

The following is the PE configuration:

pe_name           impl
slots             999
user_lists        NONE
xuser_lists       NONE
start_proc_args   NONE
stop_proc_args    NONE
allocation_rule   $round_robin
control_slaves    TRUE
job_is_first_task FALSE
urgency_slots     min

The answer would depend on how the <pe_name> parallel environment (PE) is configured. In general, -pe <pe_name> <number1> requests <number1> slots in the <pe_name> PE. Each PE could be configured to provide a fixed amount of slots on a node, or to fill up the available slots on one node and then move to the next one, to always allocate slots on the same node and so on. A slot in SGE usually corresponds to a CPU core but it is entirely to the SGE administrator to decide if this is the case or not.

-np <number2> tells Open MPI how many processes to launch within the MPI job. In many cases this number should be equal to the number of SGE slots requested. If Open MPI was built with SGE integration, it automatically gets the total number of slots granted from the batch system and explicit specifiction of the number of processes is only necessary in some special cases.

Again, it all depends on how SGE is configured. Without the details on your cluster, e.g. the output from qconf -sp <pe_name>, you won't get a very concrete answer.

mpi - Spreading a job over different nodes of a cluster in sun grid engine (SGE) - Stack Overflow

down votefavorite I'm tryin get sun gridending (sge) to run the separate processes of an MPI job over all of the nodes of my cluster.

What is happening is that each node has 12 processors, so SGE is assigning 12 of my 60 processes to 5 separate nodes.

I'd like it to assign 2 processes to each of the 30 nodes available, because with 12 processes (dna sequence alignments) running on each node, the nodes are running out of memory.

So I'm wondering if it's possible to explicitly get SGE to assign the processes to a given node?

Thanks,

Paul.

mpi sungridengine share|improve this question

3 Answers

You can do it by creating a queue in which you can define the queue uses only only 2 processors out of 12 processors in each node.

You can see configuration of current queue by using the command

qconf -sq queuename

you will see following in the queue configuration. This queue named in such a way that it uses only 5 execution hosts and 4 slots (processors) each.

....
slots                 1,[master=4],[slave1=4],[slave2=4],[slave3=4],[slave4=4]
....

use following command to change the queue configuration

qconf -mq queuename

then change those 4 into 2.

From an admin host, run "qconf -msconf" to edit the scheduler configuration. It will bring up a list of configuration options in an editor. Look for one called "load_factor". Set the value to "-slots" (without the quotes).

This tells the scheduler that the machine is least loaded when it has the fewest slots in use. If your exec hosts have a similar number of slots each, you will get an even distribution. If you have some exec hosts that have more slots than the others, they will be preferred, but your distribution will still be more even than the default value for load_factor (which I don't remember, having changed this in my cluster quite some time ago).

You may need to set the slots on each host. I have done this myself because I need to limit the number of jobs on a particular set of boxes to less than their maximum because they don't have as much memory as some of the other ones. I don't know if it is required for this load_factor configuration, but if it is, you can add a slots consumable to each host. Do this with "qconf -me hostname", add a value to the "complex_values" that looks like "slots=16" where 16 is the number of slots you want that host to use.

openmpi - Submitting Open MPI jobs to SGE - Stack Overflow

I've installed openmpi , not in /usr/... but in a /commun/data/packages/openmpi/ , it was compiled with --with-sge.

I've added a new PE in SGE as descibed in http://docs.oracle.com/cd/E19080-01/n1.grid.eng6/817-5677/6ml49n2c0/index.html

# /commun/data/packages/openmpi/bin/ompi_info | grep gridengine
MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.3)

# qconf -sq all.q | grep pe_
pe_list               make orte

Without SGE, the program runs without any problem, using several processors.

/commun/data/packages/openmpi/bin/orterun -np 20 ./a.out args

Now I want to submit my program to SGE

In the Open MPI FAQ, I read:

# Allocate a SGE interactive job with 4 slots
# from a parallel environment (PE) named 'orte'
shell$ qsh -pe orte 4

but my output is:

qsh -pe orte 4
Your job 84550 ("INTERACTIVE") has been submitted
waiting for interactive job to be scheduled ...
Could not start interactive job.

I've also tried the mpirun command embedded in a script:

$ cat ompi.sh
#!/bin/sh
/commun/data/packages/openmpi/bin/mpirun  \
    /path/to/a.out args

but it fails

$ cat ompi.sh.e84552
error: executing task of job 84552 failed: execution daemon on host "node02" didn't accept task
--------------------------------------------------------------------------
A daemon (pid 18327) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
error: executing task of job 84552 failed: execution daemon on host "node01" didn't accept task
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.

How can I fix this?

mpi openmpi sungridengine share|improve this question

edited Feb 8 '13 at 15:37

Answers

See http://www.open-mpi.org/community/lists/users/2013/02/21360.php

Open MPI User's Mailing List Archives

Subject: Re: [OMPI users] newbie: Submitting Open MPI jobs to SGE ( `qsh, -pe orte 4` fails)
From: Pierre Lindenbaum (pierre.lindenbaum_at_[hidden])
Date: 2013-02-11 06:26:17

• Next message: Daniel Fetchinson: "[OMPI users] mpirun completes for one user, not for another"
• Previous message: Jeff Squyres (jsquyres): "Re: [OMPI users] how to find the binding of each rank on the local machine"
• Next in thread: Reuti: "Re: [OMPI users] newbie: Submitting Open MPI jobs to SGE ( `qsh, -pe orte 4` fails)"
• Reply: Reuti: "Re: [OMPI users] newbie: Submitting Open MPI jobs to SGE ( `qsh, -pe orte 4` fails)"

> This is a good sign, as it tries to use `qrsh -inherit ...` already. Can you confirm the following settings:
>
> $ qconf -sp orte
> ...
> control_slaves TRUE
>
> $ qconf -sq all.q
> ...
> shell_start_mode unix_behavior
>
> -- Reuti

qconf -sp orte

pe_name orte
slots 448
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule $round_robin
control_slaves FALSE
job_is_first_task TRUE
urgency_slots min
accounting_summary FALSE

and

qconf -sq all.q | grep start_
shell_start_mode posix_compliant

I've edited the env conf using `qconf -mp orte` changing
`control_slaves` to TRUE

# qconf -sp orte
pe_name orte
slots 448
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule $round_robin
control_slaves TRUE
job_is_first_task TRUE
urgency_slots min
accounting_summary FALSE

and I've changed `shell_start_mode posix_compliant` to
`unix_behavior ` using `qconf -mconf`. (However, shell_start_mode is
still listed as posix_compliant )

Now, qsh -pe orte 4 works

qsh -pe orte 4
Your job 84581 ("INTERACTIVE") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 84581 has been successfully scheduled.

(should I run that command before running any a new mpirun command ?)

when invoking:

qsub -cwd -pe orte 7 with-a-shell.sh
or
qrsh -cwd -pe orte 100 /commun/data/packages/openmpi/bin/mpirun
/path/to/a.out arg1 arg2 arg3 ....

that works too ! Thank you ! :-)

queuename qtype resv/used/tot. load_avg
arch states
---------------------------------------------------------------------------------
all.q_at_node01 BIP 0/15/64 2.76 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 15
---------------------------------------------------------------------------------
all.q_at_node02 BIP 0/14/64 3.89 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 14
---------------------------------------------------------------------------------
all.q_at_node03 BIP 0/14/64 3.23 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 14
---------------------------------------------------------------------------------
all.q_at_node04 BIP 0/14/64 3.68 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 14
---------------------------------------------------------------------------------
all.q_at_node05 BIP 0/15/64 2.91 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 15
---------------------------------------------------------------------------------
all.q_at_node06 BIP 0/14/64 3.91 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 14
---------------------------------------------------------------------------------
all.q_at_node07 BIP 0/14/64 3.79 lx24-amd64
84598 0.55500 mpirun lindenb r 02/11/2013 12:03:36 14

OK, my first openmpi program works. But as far as I can see: it is
faster when invoked on the master node (~3.22min) than when invoked by
means of SGE (~7H45):

time /commun/data/packages/openmpi/bin/mpirun -np 7 /path/to/a.out
arg1 arg2 arg3 ....
670.985u 64.929s 3:32.36 346.5% 0+0k 16322112+6560io 32pf+0w

time qrsh -cwd -pe orte 7 /commun/data/packages/openmpi/bin/mpirun
/path/to/a.out arg1 arg2 arg3 ....
0.023u 0.036s 7:45.05 0.0% 0+0k 1496+0io 1pf+0w

I'm going to investigate this... :-)

Thank you again

Pierre

• Next message: Daniel Fetchinson: "[OMPI users] mpirun completes for one user, not for another"
• Previous message: Jeff Squyres (jsquyres): "Re: [OMPI users] how to find the binding of each rank on the local machine"
• Next in thread: Reuti: "Re: [OMPI users] newbie: Submitting Open MPI jobs to SGE ( `qsh, -pe orte 4` fails)"
• Reply: Reuti: "Re: [OMPI users] newbie: Submitting Open MPI jobs to SGE ( `qsh, -pe orte 4` fails)"

Integrating Intel MPI Library with Sun Grid Engine

12/21/2009

Gergana Slavova... on Mon, 12/21/2009 - 14:31

So, you want to use Intel® MPI Library with the Sun* Grid Engine* (SGE) batch scheduler?

The below instructions describe how to run Intel MPI Library jobs using Sun Grid Engine. This document relates to Linux*. While there are some differences and additional steps when using Microsoft* Windows*, in general the procedure is the same.

All optional steps are recommended but not necessary for successful integration.

1. [Optional] Visit sun.com and get a brief overview of SGE
2. Installation

   See the Installation Guide from sun.com for details. Roughly, the steps are as follows:

   • Install Master Host (see 'How to Install the MasterHost' section);
   • Install Execution Host (see 'How to Install ExecutionHosts' section);
   • Register Administration Hosts (see the corresponding section in the Installation Guide);
   • Register Submit Hosts (see corresponding section);
   • Verify the installation (see corresponding section).

   IMPORTANT NOTES:

   • To finalize the installation process, you'll have to configure the network services manually (by modifying /etc/services), which requires root privileges.
   • It's possible to install/run SGE as a non-privileged user, but
     1. there are some limitations in that case;
     2. you need root privileges for the complete installation process (at least, for modifying /etc/services).
3. Create a new Parallel Environment (PE) for Intel MPI Library
   
   1. Create the appropriate configuration file for the new PE. It should contain the following lines:

      pe_name	impi
      slots	999
      user_lists	NONE
      xuser_lists	NONE
      start_proc_args	NONE
      stop_proc_args	NONE
      allocation_rule	$round_robin
      control_slaves	TRUE
      job_is_first_task	FALSE
      urgency_slots	min

   2. Add the new PE using the following command:

      'qconf -Ap <config_file>'

   USEFUL COMMANDS:
   * qconf -spl – view all PEs currently available;
   * qconf -sp <PE_name> - view settings for a particular PE;
   * qconf -dp <PE_name> - remove a PE;
   * qconf -mp <PE_name> - modify an existing PE.

   Also see the 'Managing Special Environment' section in the Administration Guide from sun.com if you need more details about PE configuration.

4. Associate a queue with the new PE

   Use the following commands for that:

   1. qconf -sql – to see all queues available;
   2. qconf -mq <queue_name> - to modify the queue's settings. Find the 'pe_list' property in the open window and add the 'impi' string to that property.

   USEFUL COMMANDS:
   * qconf -sq <queue_name> - view the queue's settings.

   See the Administration Guide if you need more details about the queue configuration process.

5. Add Intel MPI Library environment to your current environment by sourcing the appropriate mpivars.[c]sh script located in the <install_dir>/bin[64] directory
6. Build the MPI application to be run
7. [Optional] Make sure that Intel MPI Library works fine on the desired hosts. For this, manually run your application on the desired hosts individually
8. Submit your MPI job to SGE

   Use the following command for that:

   qsub -N <job_name> -pe impi <num_of_processes> \
-V <mpirun_absolute_name> -r ssh -np <num_of_processes> <app_absolute_name>

   where
   -V option is used so that all environment variables available in the current shell are exported to a job.

   USEFUL COMMANDS to monitor and control jobs:
   * qstat – show status of SGE jobs and queues;
   * qstat -j – show detailed information about jobs (can be useful for pending jobs);
   * qdel – remove existing job.
   After submitting the job you can monitor its status using the qstat command. When the job is finished, you can find the job's output and error output in your HOME directory – just look for <job_name>.o<jobID> and <job_name>.e<jobID> files.

   See the User's Guide, if you need more information about the job submission process.

Closer integration with SGE

Read the 'Tight Integration of Parallel Environments and Grid Engine Software' section in SGE's Administration Guide first.

To enable tight integration for Intel MPI, use the same procedure as the one mentioned above, but use a different configuration file for the PE at step #3.

The configuration file should contain the following lines:

pe_name	impi_tight
slots	999
user_lists	NONE
xuser_lists	NONE
start_proc_args	<SGE_install_dir>/mpi/startmpi.sh -catch_rsh $pe_hostfile
stop_proc_args	<SGE_install_dir>/mpi/stopmpi.sh
allocation_rule	$round_robin
control_slaves	TRUE
job_is_first_task	FALSE
urgency_slots	min

FAQ Running jobs under SGE

OpenMPI FAQ

1. How do I run with the SGE launcher?

Support for SGE is included in Open MPI version 1.2 and later.

NOTE: To build SGE support in v1.3, you will need to explicitly request the SGE support with the "--with-sge" command line switch to Open MPI's configure script.

See this FAQ entry for a description of how to correctly build Open MPI with SGE support.

To verify if support for SGE is configured into your Open MPI installation, run ompi_info as shown below and look for gridengine. The components you will see are slightly different between v1.2 and v1.3.

For Open MPI 1.2:

shell$ ompi_info | grep gridengine
                 MCA ras: gridengine (MCA v1.0, API v1.0, Component v1.2)
                 MCA pls: gridengine (MCA v1.0, API v1.0, Component v1.2)

For Open MPI 1.3:

shell$ ompi_info | grep gridengine
                 MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.3)

Open MPI will automatically detect when it is running inside SGE and will just "do the Right Thing."

Specifically, if you execute an mpirun command in a SGE job, it will automatically use the SGE mechanisms to launch and kill processes. There is no need to specify what nodes to run on -- Open MPI will obtain this information directly from SGE and default to a number of processes equal to the slot count specified. For example, this will run 4 MPI processes on the nodes that were allocated by SGE:

# Get the environment variables for SGE
# (Assuming SGE is installed at /opt/sge and $SGE_CELL is 'default' in your environment)
# C shell settings
shell% source /opt/sge/default/common/settings.csh

# bourne shell settings
shell$ . /opt/sge/default/common/settings.sh

# Allocate an SGE interactive job with 4 slots from a parallel
# environment (PE) named 'orte' and run a 4-process Open MPI job
shell$ qrsh -pe orte 4 -b y mpirun -np 4 a.out

There are also other ways to submit jobs under SGE:

# Submit a batch job with the 'mpirun' command embedded in a script
shell$ qsub -pe orte 4 my_mpirun_job.csh

# Submit an SGE and OMPI job and mpirun in one line
shell$ qrsh -V -pe orte 4 mpirun hostname

# Use qstat(1) to show the status of SGE jobs and queues
shell$ qstat -f

In reference to the setup, be sure you have a Parallel Environment (PE) defined for submitting parallel jobs. You don't have to name your PE "orte". The following example shows a PE named 'orte' that would look like:

% qconf -sp orte
   pe_name            orte
   slots              99999
   user_lists         NONE
   xuser_lists        NONE
   start_proc_args    NONE
   stop_proc_args     NONE
   allocation_rule    $fill_up
   control_slaves     TRUE
   job_is_first_task  FALSE
   urgency_slots      min
   accounting_summary FALSE
   qsort_args         NONE

"qsort_args" is necessary with the Son of Grid Engine distribution, version 8.1.1 and later, and probably only applicable to it. For very old versions of SGE, omit "accounting_summary" too.

You may want to alter other parameters, but the important one is "control_slaves", specifying that the environment has "tight integration". Note also the lack of a start or stop procedure. The tight integration means that mpirun automatically picks up the slot count to use as a default in place of the '-np' argument, picks up a host file, spawns remote processes via 'qrsh' so that SGE can control and monitor them, and creates and destroys a per-job temporary directory ($TMPDIR), in which Open MPI's directory will be created (by default).

Be sure the queue will make use of the PE that you specified:

% qconf -sq all.q
...
pe_list               make cre orte
...

To determine whether the SGE parallel job is successfully launched to the remote nodes, you can pass in the MCA parameter "--mca plm_base_verbose 1" to mpirun.

This will add in a -verbose flag to qrsh -inherit command that is used to send parallel tasks to the remote SGE execution hosts. It will show whether the connections to the remote hosts are established successfully or not.

Various SGE documentation with pointers to more is available at the Son of GridEngine site, and configuration instructions can be found at the Son of GridEngine configuration how-to site..

Sun Grid Engine (SGE) QuickStart

OpenMPI and Sun Grid Engine¶

Note

OpenMPI must be compiled with SGE support (–with-sge) to make use of the tight-integration between OpenMPI and SGE as documented in this section.

OpenMPI supports tight integration with Sun Grid Engine. This integration allows Sun Grid Engine to handle assigning hosts to parallel jobs and to properly account for parallel jobs.

OpenMPI Parallel Environment¶

StarCluster by default sets up a parallel environment, called "orte", that has been configured for OpenMPI integration within SGE and has a number of slots equal to the total number of processors in the cluster. You can inspect the SGE parallel environment by running:

sgeadmin@ip-10-194-13-219:~$ qconf -sp orte
pe_name            orte
slots              16
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $round_robin
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE

This is the default configuration for a two-node, c1.xlarge cluster (16 virtual cores).

Round Robin vs Fill Up Modes¶

Notice the allocation_rule setting in the output of the qconf command in the previous section. This defines how to assign slots to a job. By default StarCluster configures round_robin allocation. This means that if a job requests 8 slots for example, it will go to the first machine, grab a single slot if available, move to the next machine and grab a single slot if available, and so on wrapping around the cluster again if necessary to allocate 8 slots to the job.

You can also configure the parallel environment to try and localize slots as much as possible using the fill_up allocation rule. With this rule, if a user requests 8 slots and a single machine has 8 slots available, that job will run entirely on one machine. If 5 slots are available on one host and 3 on another, it will take all 5 on that host, and all 3 on the other host. In other words, this rule will greedily take all slots on a given node until the slot requirement for the job is met.

You can switch between round_robin and fill_up modes using the following command:

$ qconf -mp orte

This will open up vi (or any editor defined in EDITOR env variable) and let you edit the parallel environment settings. To change from round_robin to fill_up in the above example, change the allocation_rule line from:

allocation_rule    $round_robin

to:

allocation_rule    $fill_up

After making the change and saving the file you can verify your settings using:

sgeadmin@ip-10-194-13-219:~$ qconf -sp orte
pe_name            orte
slots              16
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $fill_up
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary FALSE

Submitting OpenMPI Jobs using a Parallel Environment¶

The general workflow for running MPI code is:

1. Compile the code using mpicc, mpicxx, mpif77, mpif90, etc.
2. Copy the resulting executable to the same path on all nodes or to an NFS-shared location on the master node

Note

It is important that the path to the executable is identical on all nodes for mpirun to correctly launch your parallel code. The easiest approach is to copy the executable somewhere under /home on the master node since /home is NFS-shared across all nodes in the cluster.

3. Run the code on X number of machines using:

   $ mpirun -np X -hostfile myhostfile ./mpi-executable arg1 arg2 [...]

where the hostfile looks something like:

$ cat /path/to/hostfile
master  slots=2
node001 slots=2
node002 slots=2
node003 slots=2

However, when using an SGE parallel environment with OpenMPI you no longer have to specify the -np, -hostfile, -host, etc. options to mpirun. This is because SGE will automatically assign hosts and processors to be used by OpenMPI for your job. You also do not need to pass the -byslot and -bynode options to mpirun given that these mechanisms are now handled by the fill_up and round_robin modes specified in the SGE parallel environment.

Instead of using the above formulation create a simple job script that contains a very simplified mpirun call:

$ cat myjobscript.sh
mpirun /path/to/mpi-executable arg1 arg2 [...]

Then submit the job using the qsub command and the orte parallel environment automatically configured for you by StarCluster:

$ qsub -pe orte 24 ./myjobscript.sh

The -pe option species which parallel environment to use and how many slots to request. The above example requests 24 slots (or processors) using the orte parallel environment. The parallel environment automatically takes care of distributing the MPI job amongst the SGE nodes using the allocation_rule defined in the environment's settings.

You can also do this without a job script like so:

$ cd /path/to/executable
$ qsub -b y -cwd -pe orte 24 mpirun ./mpi-executable arg1 arg2 [...]

Configuring a New Parallel Environment

Mar 23, 2007 | DanT's Grid Blog

By templedf on Mar 23, 2007

Since this seems to be a regular topic on the user mailing list, here's a quick guide to setting up a parallel environment on Grid Engine:

1. First, create/borrow/steal the startup and shutdown scripts for the parallel environment you're using. You can find MPI and PVM scripts in the $SGE_ROOT/mpi and $SGE_ROOT/pvm directories, respectively. If you cannot find scripts for your parallel environment, you'll have to create them. The startup script must prepacdre the parallel environment for being used. With most MPI implementations, that's just a matter of creating a "machines" file that lists the machines which are to run the parallel job. The shutdown script must clean up after the parallel job's execution. The MPI shutdown script just deletes the "machines" file.
   
2. Next, you have to tell Grid Engine about your parallel environment. You can do that interactively with qmon or qconf -ap <pe_name> or you can write the data to a file and use qconf -Ap <file_name>. For an example of what such a file would look like, see $SGE_ROOT/mpi/mpi.template or $SGE_ROOT/pvm/pvm.template.

   Let's look at what the parallel environment configuration contains.

   pe_name           template
   slots             0
   user_lists        NONE
   xuser_lists       NONE
   start_proc_args   /bin/true
   stop_proc_args    /bin/true
   allocation_rule   $pe_slots
   control_slaves    FALSE
   job_is_first_task FALSE
   urgency_slots     min

   • pe_name - the name by which the parallel environment will be known to Grid Engine
   • slots - the maximum number of job slots that the parallel environment is allowed to occupy at once
   • users_lists - an ACL specifying the users who are allowed to use the parallel environment. If set to NONE, it means any user can use it
   • xusers_list - an ACL specifying the users who are not allowed to use the parallel environment. Users in both the users_list and xusers_list are not allowed to use the parallel environment
   • start_proc_args - the path to the startup script for the parallel environment followed by any needed arguments. Grid Engine provides some inline variables that you can use as arguments:
     • $pe_hostfile - the path to a file written by Grid Engine which contains information about how and where the parallel job should be run
     • $host - the host on which the parallel environment is being started
     • $job_owner - the name of the user who owns the parallel job
     • $job_id - the id of the parellel job
     • $job_name - the name of the parallel job
     • $pe - the name of the parallel environment
     • $pe_slots - the number of job slots assigned to the job
     • $queue - the name of the queue in which the parallel job is running

     The value of this setting is the command that will be run to start the parallel environment for every parallel job.

   • stop_proc_args - the path to the shutdown script for the parallel environment followed by any needed arguments. The same inline variables are available as with start_proc_args.
   • allocation_rule - this setting controls how job slots are assigned to hosts. It can have four possible values:
     • a number - if set to a number, Grid Engine will assign that many slots to the parallel job on each host until the assigned number of job slots is met. Setting this attribute to 1, for example, would mean that the job gets a single job slot on each host where it is assigned. Grid Engine will not assign the job more job slots than the number of assigned hosts multiplied by this attribute's value.
     • $fill_up - use all of the job slots on a given host before moving to the next host
     • $round_robin - select one slot from each host in a round-robin fashion until all job slots are assigned. This setting can result in more than one job slot per host.
     • $pe_slots - place all the job slots on a single machine. Grid Engine will only schedule such a job to a machine that can host the maximum number of slots requested by the job. (See below.)
   • control_slaves - this setting tells Grid Engine whether the parallel environment integration is "tight" or "loose". See your parallel environment's documentation for more details.
   • job_is_first_task - this setting tells Grid Engine whether the first task of the parallel job is actually a job task or whether it's just there to kick off the rest of the jobs. This setting is also determined by your parallel environment integration.
   • urgency_slots - this setting affect how resource requests affect job priority for parallel jobs. The values can be "min," "max," "avg," or a number. For more information about resource-based job priorities, see this white paper

   For more information about these settings, see the sge_pe man page.
   

3. The next step is to enable your parallel environment for the queues where it should be available. You can add the parallel environment to a queue interactively with qmon or qconf -mq <queue> or in a single action with qconf -aattr queue pe_list <pe_name> <queue>.
   
4. Now you're ready to test your parallel environment. Run qsub -pe <pe_name> <slots>. Aside from the usual output and error files (<job_name>.o<job_id> and <job_name>.e<job_id>, respectively), you should also look for the parallel environment startup output and error files, <job_name>.po<job_id> and <job_name>.pe<job_id>.

That's all there is to it! Just to make sure we're clear on everything, let's do an example. Let's create a parallel environment that starts up an RMI registry and stores the port number in a file so that the job can find it.

First thing we have to do is write the startup and shudown scripts for the RMI parallel environment. Here's what they look like:

rmi_startup.sh

#!/bin/sh
# $TMPDIR and $JOB_ID are set by Grid Engine automatically

# Borrowed from $SGE_ROOT/mpi/startmpi.sh
PeHostfile2MachineFile()
{
   cat $1 | while read line; do
      host=`echo $line|cut -f1 -d" "|cut -f1 -d"."`
      nslots=`echo $line|cut -f2 -d" "`
      i=1

      while [ $i -le $nslots ]; do
         echo $host
         i=`expr $i + 1`
      done
   done
}

# get arguments
pe_hostfile=$1

# ensure pe_hostfile is readable
if [ ! -r $pe_hostfile ]; then
   echo "$me: can't read $pe_hostfile" >&2
   exit 1
fi

# create machines file
machines="$TMPDIR/machines"
PeHostfile2MachineFile $pe_hostfile >> $machines

# We use ports 40000-40999
port=`expr \\( $JOB_ID % 1000 \\) + 40000`

# Start the registry
/usr/java/bin/rmiregistry $port &

# Save the registry's PID so that we can stop it later
echo $! > $TMPDIR/pid

# Save the port number so the job can find it
echo $port > $TMPDIR/port

rmi_shutdown.sh

#!/bin/sh
# $TMPDIR is set by Grid Engine automatically

# Get the registry's PID
pid=`cat $TMPDIR/pid`

# Kill the registry
kill $pid

# Clean up the files the startup script created
rm $TMPDIR/pid
rm $TMPDIR/port
rm $TMPDIR/machines

Next thing we have to do is add our parallel environment to Grid Engine. First we create a file, say /tmp/rmi_pe, with the following contents:

pe_name           rmi
slots             4
user_lists        NONE
xuser_lists       NONE
start_proc_args   /home/dant/rmi_startup.sh $pe_hostfile
stop_proc_args    /home/dant/rmi_shutdown.sh
allocation_rule   $round_robin
control_slaves    TRUE
job_is_first_task FALSE
urgency_slots     min

Note that control_slaves is true and job_is_first_task is false. Because we're writing the integration scripts, the choice is somewhat arbitrary, but it affects how the job scripts must be written, as we'll see below. It also affect whether the qmaster is able to keep accounting records on the slave tasks. If control_slaves is false, the qmaster is have no records of how much resources the slaves tasks consumed.

Now we add the parallel environment with qconf -Ap /tmp/rmi_pe. We could have skipped a step by running qconf -ap rmi and entering the data in the editor that comes up, but they way we've done it here is scriptable.

The next step is to add our parallel environment to our queue with qconf -aattr queue pe_list rmi all.q. Again, we could have run qconf -mq all.q and edited the pe_list attribute in the editor, but the way we've done it is scriptable.

Last thing to do is test out our parallel environment. First we need a job script:

#!/bin/sh
#$ -S /bin/sh

port=`cat $TMPDIR/port`
qrsh=$SGE_ROOT/bin/$ARC/qrsh

cat $TMPDIR/machines | while read host; do
   $qrsh -inherit $host /usr/bin/java -cp ~/rmi.jar RMIApp $port &
done

Let's look at this job script for a moment. The first thing to notice is the use of qrsh -inherit. The -inherit switch is specifically for kicking off slave tasks. It requires that the target host name be supplied. In order to get the target host name, we read the machines file that the startup script generated from the one Grid Engine supplied.

The second thing to notice is how ugly the use of qrsh -inherit is. RMI is not really a parallel environment. It's a communications framework. It doesn't do the work of kicking off remote processes for you. So, instead, we have to do it ourselves in the job script. With a true parallel environment, like any of the MPI flavors, the framework also takes care of starting the remote processes, often through rsh. In the MPI scripts included with Grid Engine, an rsh wrapper script is included, which transparently replaces calls to rsh with calls to qrsh -inherit. By using that wrapper script, the parallel environment's calls to rsh can be rerouted through the grid via qrsh without having to modify the parallel environment itself to work with Grid Engine.

The last thing to notice is how this script correlates to the control_slaves and job_is_first_task attributes of the parallel environment configuration. Let's start with first_job_is_task. In our configuration, we set it to false. That means that this master job script is not counted as a job and does no real work. That is why our script doesn't do anything but kick off sub-tasks. If job_is_first_task had been true, our job script would be expected to run one of the RMIApp instances itself.

Now let's talk about the control_slaves attribute. If control_slaves is true, we are allowed to use qrsh -inherit to kick off our sub-tasks. The qmaster will not, however, allow us to kick off more subtasks than the number of slots we've been assigned (minus 1 if job_is_first_task is true). The advantage of using qrsh -inherit is that the sub-tasks are tracked by Grid Engine like regular jobs. If control_slaves is false, we have to use some mechanism external to Grid Engine, such as rsh or ssh, to kick off our sub-tasks, meaning that Grid Engine cannot track them and is actually fully unaware of them. That's why job_is_first_task is meaningless when control_slaves is false.

In order to test our job we need a Java application called RMIApp. As that's outside the scope of the example, let's just pretend we have a parallel Java application that uses the RMI registry for intra-process communication. To submit our job we use qsub -pe rmi 2-4 rmi_job.sh. The -pe rmi 2-4 argument tells the qmaster that we're using the rmi parallel environment and we want 4 job slots assigned to our job, but we will accept as few as 2. Because our job script starts a sub-task for every entry in to host file, it will start the right number of sub-tasks, no matter how many slots we are assigned. Had we written the job script to start exactly two sub-tasks, we would have to use -pe rmi 2 so that we could be sure we got exactly two job slots.

While the job is running, run qstat -f. You'll see output something like this:

% qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@ultra20                  BIP   4/10      0.08     sol-amd64
    253 0.55500 rmi_job.sh dant         r     03/23/2007 11:46:51     4

From this output we can see that the job has been scheduled and has been assigned four job slots. Those four job slots only account for the four sub-tasks. The master job itself is not counted because the job_is_first_task attribute is false.

After our job completes, if we look in our home directory (which is where Grid Engine will put the output files since we didn't tell it otherwise), we will find four new files: rmi_job.sh.e253, rmi_job.sh.o253, rmi_job.sh.pe253, and rmi_job.sh.o253, assuming, of course, that our job was number 253. The \*.o253 and \*.e253 files should be familiar. They're the output and error streams from the job script. The \*.po253 and \*.pe253 files are new. They're the output and error streams from the parallel environment startup and shutdown scripts.

So, there you have it. A complete, top-to-bottom example of creating, configuring, and running a parallel environment.

clustering.gridengine.users - Re pe_slots issues - msg#00048

OSDir.com

On Thursday 01 November 2007 17:43, Reuti wrote:
> Am 01.11.2007 um 21:16 schrieb John Coldrick:
> > I've got a PE set up:
> > ***
> > pe_name m9_1
> > slots 999
> > user_lists NONE
> > xuser_lists NONE
> > start_proc_args /bin/true
> > stop_proc_args /bin/true
> > allocation_rule $pe_slots
>
> You might want to try $round_robin All possible values are listed in
> man sge_pe.

Thanks for getting back...my apps can't run cross-system, they all must run on a single system, which is why I'm using $pe_slots. I assume that's the one I should be using, right? If I do use round_robin, it splits the 4 slots up over multiple systems, as it should.

> No. With $pe_slots all slots must come from one node, and maybe there
> is already something else running, so you can't get more. You defined
> 8 also for the slot count in the queue configuration for these three
> nodes?

Correct. I've got complete control of the grid, so nothing else is running when I'm testing, and all the systems have all their slots open and available.

> Often advisable with parallel jobs is to request reservation with "-R
> y" in qsub and set a sensible value for "max_reservation" in the
> scheduler configuration.

That makes no difference - I've tried reserving along with variations of 0-8 for the max reservation, and the behaviour is the same.

Is there anywhere else where a maximum ceiling of three(or 'n') slots could exist? SGE6.0 worked fine with this, I like to keep current if I can, though. :) I can't help but think there's a new default somewhere that
didn't exist in 6.0 that's I'm getting caught up on. Just to check - getting that message that the PE only offers '0' slots - isn't that indicative of something being very wrong? If I qalter the existing job to 3 slots, off it
goes, it runs using the PE. It seems fundamentally wrong to me that this message shows up at all given that, unless it's more generic than I'm assuming and it's a catchall for numerous variables failing, such as load or
mem(which, btw, is fine, provable by requesting 3 slots running fine).

Any suggestions appreciated...

Cheers,

J.C.
--
John Coldrick www.axyzfx.com Axyz Animation
416-504-0425 477 Richmond St W
Toronto, ON Canada
jc@xxxxxxxxxx M5V 3E7
-----------------------------------------------------------------------
"Life is too important to take seriously."
-- Corky Siegel

Recommended Links

Softpanorama Recommended

Top articles

Sites

FAQ Running jobs under SGE

Tight MPICH Integration in Grid Engine

• Ubuntu Manpage sge_pe - Sun Grid Engine parallel environment configuration file format
• Ubuntu Manpage sge_types - Sun Grid Engine type descriptions
• Ubuntu Manpage sge_types - Sun Grid Engine type descriptions

pe_name
A "pe_name" is the name of a Sun Grid Engine parallel environment
described in sge_pe(5).

pe_name := object_name

Grid Engine HOWTOs

Reference

Parallel environments are parallel programming and runtime environments allowing for the execution of shared memory or distributed memory parallelized applications. Parallel environments usually require some kind of setup to be operational before starting parallel applications. Examples for common parallel environments are shared memory parallel operating systems and the distributed memory environments Parallel Virtual Machine (PVM) or Message Passing Interface (MPI).

sge_pe allows for the definition of interfaces to arbitrary parallel environments. Once a parallel environment is defined or modified with the -ap or -mp options to qconf(1) and linked with one or more queues via pe_list in queue_conf(5) the environment can be requested for a job via the -pe switch to qsub(1) together with a request of a range for the number of parallel processes to be allocated by the job. Additional -l options may be used to specify the job requirement to further detail.

Note, Sun Grid Engine allows backslashes (\) be used to escape newline (\newline) characters. The backslash and the newline are replaced with a space (" ") character before any interpretation.

Format

The format of a sge_pe file is defined as follows:

pe_name

The name of the parallel environment as defined for pe_name in sge_types(1). To be used in the qsub(1) -pe switch.

slots

The number of parallel processes being allowed to run in total under the parallel environment concurrently. Type is number, valid values are 0 to 9999999.

user_lists

A comma separated list of user access list names (see access_list(5)). Each user contained in at least one of the enlisted access lists has access to the parallel environment. If the user_lists parameter is set to NONE (the default) any user has access being not explicitly excluded via the xuser_lists parameter described below. If a user is contained both in an access list enlisted in xuser_lists and user_lists the user is denied access to the parallel environment.

xuser_lists

The xuser_lists parameter contains a comma separated list of so called user access lists as described in access_list(5). Each user contained in at least one of the enlisted access lists is not allowed to access the parallel environment. If the xuser_lists parameter is set to NONE (the default) any user has access. If a user is contained both in an access list enlisted in xuser_lists and user_lists the user is denied access to the parallel environment.

start_proc_args

The invocation command line of a start-up procedure for the parallel environment. The start-up procedure is invoked by sge_shepherd(8) prior to executing the job script. Its purpose is to setup the parallel environment correspondingly to its needs. An optional prefix "user@" specifies the user under which this procedure is to be started. The standard output of the start-up procedure is redirected to the file REQNAME.poJID in the job's working directory (see qsub(1)), with REQNAME being the name of the job as displayed by qstat(1) and JID being the job's identification number. Likewise, the standard error output is redirected to REQNAME.peJID
The following special variables being expanded at runtime can be used (besides any other strings which have to be interpreted by the start and stop procedures) to constitute a command line:

$pe_hostfile

The pathname of a file containing a detailed description of the layout of the parallel environment to be setup by the start-up procedure. Each line of the file refers to a host on which parallel processes are to be run. The first entry of each line denotes the hostname, the second entry the number of parallel processes to be run on the host, the third entry the name of the queue, and the fourth entry a processor range to be used in case of a multiprocessor machine.

$host

The name of the host on which the start-up or stop procedures are started.

$job_owner

The user name of the job owner.

$job_id

Sun Grid Engine's unique job identification number.

$job_name

The name of the job.

$pe

The name of the parallel environment in use.

$pe_slots

Number of slots granted for the job.

$processors

The processors string as contained in the queue configuration (see queue_conf(5)) of the master queue (the queue in which the start-up and stop procedures are started).

$queue

The cluster queue of the master queue instance.

stop_proc_args

The invocation command line of a shutdown procedure for the parallel environment. The shutdown procedure is invoked by sge_shepherd(8) after the job script has finished. Its purpose is to stop the parallel environment and to remove it from all participating systems. An optional prefix "user@" specifies the user under which this procedure is to be started. The standard output of the stop procedure is also redirected to the file REQNAME.poJID in the job's working directory (see qsub(1)), with REQNAME being the name of the job as displayed by qstat(1) and JID being the job's identification number. Likewise, the standard error output is redirected to REQNAME.peJID
The same special variables as for start_proc_args can be used to constitute a command line.

allocation_rule

The allocation rule is interpreted by the scheduler thread and helps the scheduler to decide how to distribute parallel processes among the available machines. If, for instance, a parallel environment is built for shared memory applications only, all parallel processes have to be assigned to a single machine, no matter how much suitable machines are available. If, however, the parallel environment follows the distributed memory paradigm, an even distribution of processes among machines may be favorable.
The current version of the scheduler only understands the following allocation rules:

<int>:

An integer number fixing the number of processes per host. If the number is 1, all processes have to reside on different hosts. If the special denominator $pe_slots is used, the full range of processes as specified with the qsub(1) -pe switch has to be allocated on a single host (no matter which value belonging to the range is finally chosen for the job to be allocated).

$fill_up:

Starting from the best suitable host/queue, all available slots are allocated. Further hosts and queues are "filled up" as long as a job still requires slots for parallel tasks.

$round_robin:

From all suitable hosts a single slot is allocated until all tasks requested by the parallel job are dispatched. If more tasks are requested than suitable hosts are found, allocation starts again from the first host. The allocation scheme walks through suitable hosts in a best-suitable-first order.

control_slaves

This parameter can be set to TRUE or FALSE (the default). It indicates whether Sun Grid Engine is the creator of the slave tasks of a parallel application via sge_execd(8) and sge_shepherd(8) and thus has full control over all processes in a parallel application, which enables capabilities such as resource limitation and correct accounting. However, to gain control over the slave tasks of a parallel application, a sophisticated PE interface is required, which works closely together with Sun Grid Engine facilities. Such PE interfaces are available through your local Sun Grid Engine support office.

Please set the control_slaves parameter to false for all other PE interfaces.

job_is_first_task

The job_is_first_task parameter can be set to TRUE or FALSE. A value of TRUE indicates that the Sun Grid Engine job script already contains one of the tasks of the parallel application (the number of slots reserved for the job is the number of slots requested with the -pe switch), while a value of FALSE indicates that the job script (and its child processes) is not part of the parallel program (the number of slots reserved for the job is the number of slots requested with the -pe switch + 1).

If wallclock accounting is used (execd_params ACCT_RESERVED_USAGE and/or SHARETREE_RESERVED_USAGE set to TRUE) and control_slaves is set to FALSE, the job_is_first_task parameter influences the accounting for the job: A value of TRUE means that accounting for cpu and requested memory gets multiplied by the number of slots requested with the -pe switch, if job_is_first_task is set to FALSE, the accounting information gets multiplied by number of slots + 1.

urgency_slots

For pending jobs with a slot range PE request the number of slots is not determined. This setting specifies the method to be used by Sun Grid Engine to assess the number of slots such jobs might finally get.

The assumed slot allocation has a meaning when determining the resource-request-based priority contribution for numeric resources as described in sge_priority(5) and is displayed when qstat(1) is run without -g t option.

The following methods are supported:

<int>:

The specified integer number is directly used as prospective slot amount.

min:

The slot range minimum is used as prospective slot amount. If no lower bound is specified with the range 1 is assumed.

max:

The of the slot range maximum is used as prospective slot amount. If no upper bound is specified with the range the absolute maximum possible due to the PE's slots setting is assumed.

avg:

The average of all numbers occurring within the job's PE range request is assumed.

accounting_summary

This parameter is only checked if control_slaves (see above) is set to TRUE and thus Sun Grid Engine is the creator of the slave tasks of a parallel application via sge_execd(8) and sge_shepherd(8). In this case, accounting information is available for every single slave task started by Sun Grid Engine.

The accounting_summary parameter can be set to TRUE or FALSE. A value of TRUE indicates that only a single accounting record is written to the accounting(5) file, containing the accounting summary of the whole job including all slave tasks, while a value of FALSE indicates an individual accounting(5) record is written for every slave task, as well as for the master task.
Note: When running tightly integrated jobs with SHARETREE_RESERVED_USAGE set, and with having accounting_summary enabled in the parallel environment, reserved usage will only be reported by the master task of the parallel job. No per parallel task usage records will be sent from execd to qmaster, which can significantly reduce load on qmaster when running large tightly integrated parallel jobs.

Restrictions

Note, that the functionality of the start-up, shutdown and signaling procedures remains the full responsibility of the administrator configuring the parallel environment. Sun Grid Engine will just invoke these procedures and evaluate their exit status. If the procedures do not perform their tasks properly or if the parallel environment or the parallel application behave unexpectedly, Sun Grid Engine has no means to detect this.

See Also

sge_intro(1), sge__types(1), qconf(1), qdel(1), qmod(1), qsub(1), access_list(5), sge_qmaster(8), sge_shepherd(8).

Copyright

See sge_intro(1) for a full statement of rights and permissions.

Referenced By

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: May 16, 2021