MPI startup(): RLIMIT_MEMLOCK too small problem with SGE and Infiniband

The problem does not depend on which version of MPI you are using: it is observable both with Intel MPI 4.1, Melllanoix OpenMPI 1.6.4 and several others.

One sthing you can do is to increase the RLIMIT_MEMLOCK limit to 32MB instead of just 32 or 64KB by adding the command

ulimit -l 32000

to the sgeexecd init script.

Putting

execd_params H_MEMORYLOCKED=UNLIMITED

does not work either in Univa SGE. But it works in Oracle SGE.

Top updates

Bulletin

Latest

Past week

Past month

Google Search

NEWS CONTENTS

Old News ;-)

5 messages in net.sunsource.gridengine.usersRe: [GE users] OpenMPI RLIMIT_MEMLOCK...

From Sent On Attachments

Prentice Bisbal Dec 3, 2008 12:54 pm

Prentice Bisbal Dec 3, 2008 2:06 pm

Prentice Bisbal Dec 3, 2008 2:49 pm

pollinger Dec 4, 2008 1:57 am

Prentice Bisbal Dec 4, 2008 6:34 am

Subject: Re: [GE users] OpenMPI RLIMIT_MEMLOCK problem

From: Prentice Bisbal (pren...@ias.edu)

Date: Dec 3, 2008 2:49:30 pm

List: net.sunsource.gridengine.users

I think I found a bug in 6.2. When I set
execd_params H_MEMORYLOCKED=unlimited

and restart sge_execd on all my nodes, I get this:

$ qrsh ulimit -l 4

When I set

execd_params H_MEMORYLOCKED=UNLIMITED,

I get this

$ qrsh ulimit -l 5

If set H_MEMORYLOCKED to a numerical value, like this

execd_params H_MEMORYLOCKED=32M

I get the correct result:

qrsh ulimit -l 32768

Looks like the unlimited keyword is not defined properly in the code somewhere. Also, I have to restart sge_execd to get these changes to take effect. Is that the proper behavior? The documentation I linked to in my original post didn't explicitly say anything about restarting sge_execd after tinkering with these settings.

-- Prentice

Prentice Bisbal wrote:
I posted this earlier, but it never showed up on the list. Probably b/c I unsubscribed before SC08 and forget to resubscribe. If if the first attempt shows up, I apologize for posting twice.
I'm using SGE 6.2 with OpenMPI 1.2.8.

I just setup OpenMPI tight integration following the instructions here:

http://www.open-mpi.org/faq/?category=running#run-n1ge-or-sge

I then defined my exec_params to set H_MEMLOCK=unlimited as described here:

http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38 &dsMessageId=72405

Unfortunately, when I submit an MPI job, I still get MEMLOCK errors:

libibverbs: Warning: RLIMIT_MEMLOCK is 11162 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 11199 bytes. This will severely limit memory registrations. libibverbs: Warning: RLIMIT_MEMLOCK is 11181 bytes. This will severely limit memory registrations.

To make sure my parameters were took effect, I even stopped and restarted sge_execd on every compute node with no luck.

Any ideas? Relevant configuration information is below. Please note that in qconf I've tried

1. execd_params H_MEMORYLOCKED=ulimited 2. execd_params S_MEMORYLOCKED=ulimited 3. execd_params S_MEMORYLOCKED=ulimited H_MEMORYLOCKED=ulimited

All 3 had the same result.

Let me know if additional/complete config information would be helpful. I don't want to flood the list with unnecessary config information.

# qconf -sp orte pe_name orte slots 512 user_lists NONE xuser_lists NONE start_proc_args /bin/true stop_proc_args /bin/true allocation_rule $fill_up control_slaves TRUE job_is_first_task FALSE urgency_slots min accounting_summary FALSE

# qconf -sq all.q | grep pe_list pe_list make orte

# qconf -sconf | grep execd_params execd_params H_MEMORYLOCKED=unlimited S_MEMORYLOCKED=unlimited

$ more xhpl.sh #$ -S /bin/bash #$ -N xhpl #$ -pe orte 512 #$ -cwd #$ -V

MPI=/usr/local/openmpi/gcc-4.1.2/x86_64/ PATH=${MPI}/bin:$i{PATH} LD_LIBRARY_PATH=${MPI}/lib

mpirun ./xhpl
-- Prentice

------------------------------------------------------ http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=90969

To unsubscribe from this discussion, e-mail:
[user...@gridengine.sunsource.net].

From	Sent On	Attachments
Prentice Bisbal	Dec 3, 2008 12:54 pm
Prentice Bisbal	Dec 3, 2008 2:06 pm
Prentice Bisbal	Dec 3, 2008 2:49 pm
pollinger	Dec 4, 2008 1:57 am
Prentice Bisbal	Dec 4, 2008 6:34 am

Subject:	Re: [GE users] OpenMPI RLIMIT_MEMLOCK problem
From:	Prentice Bisbal (pren...@ias.edu)
Date:	Dec 3, 2008 2:49:30 pm
List:	net.sunsource.gridengine.users

MPI startup() ofa fabric is not available

Etc

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Softpanorama May the source be with you, but remember the KISS principle ;-)	Home	Switchboard	Unix Administration	Red Hat	TCP/IP Networks	Neoliberalism	Toxic Managers
	(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix

Top updates <p>Your browser does not support iframes.</p>
Bulletin	Latest	Past week	Past month	Google Search