Job or Queue Reported in Error State E

Unreachable or error in node problem. Not a permissions problem? Well, maybe the nodes or the queues are unreachable. Check with:

qstat -f

or, for even more detail:

qstat -F

If the "state" column in qstat -f  has a big E, that host or queue is in an error state due to... well, something. Sometimes an error just occurs and marks the whole queue as "bad", which blocks all jobs from running in that queue, even though there is nothing otherwise wrong with it.

Use qmod -c <queue list>  to clear the error state for a queue.

[0]root@mysite17: # qmod -c m12a.q m32a.q
root@mysite17 changed state of "m12a.q@mysite52" (no error)
Queue instance "m32a.q@mysite16" is already in the specified state: no error
---sge server: /sge/default/spool/qmaster

[1] root@mysite17: # qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
all.q@mysite16                 BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite17                 BIP   0/0/12         0.06     lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite52                 BIP   0/0/12         12.02    lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite53                 BIP   0/0/80         39.72    lx24-amd64
---------------------------------------------------------------------------------
all.q@mysite54                 BIP   0/0/80         0.02     lx24-amd64
---------------------------------------------------------------------------------
all.q@wx3481-ustc              BIP   0/0/8          -NA-     lx24-amd64    au
---------------------------------------------------------------------------------
c12.q@mysite52                 BIP   0/0/12         12.02    lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite16                 BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite53                 BIP   0/0/32         39.72    lx24-amd64
---------------------------------------------------------------------------------
c32.q@mysite54                 BIP   0/0/32         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
c40.q@mysite53                 BIP   0/0/40         39.72    lx24-amd64
---------------------------------------------------------------------------------
c40.q@mysite54                 BIP   0/0/40         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
m12a.q@mysite52                BIP   0/12/12        12.02    lx24-amd64
---------------------------------------------------------------------------------
m32a.q@mysite16                BIP   0/0/32         0.00     lx24-amd64
---------------------------------------------------------------------------------
m40a.q@mysite54                BIP   0/0/40         0.02     lx24-amd64    E
---------------------------------------------------------------------------------
m40b.q@mysite53                BIP   0/40/40        39.72    lx24-amd64
 
# qmod -c c40.q c32.q m40a.q
root@mysite17 changed state of "c40.q@mysite54" (no error)
Queue instance "c40.q@mysite53" is already in the specified state: no error
Queue instance "c32.q@mysite16" is already in the specified state: no error
root@mysite17 changed state of "c32.q@mysite54" (no error)
Queue instance "c32.q@mysite53" is already in the specified state: no error
root@mysite17 changed state of "m40a.q@mysite54" (no error)
		

Maybe that's not the problem, though. Maybe there is some network problem preventing the SGE master from communicating with the exec hosts, such as routing problems or a firewall misconfiguration.

You can troubleshoot these things with qping, which will test whether the SGE processes on the master node and the exec nodes can communicate.

N.B.: remember, the execd  process on the exec node is responsible for establishing a TCP/IP connection to the qmaster  process on the master node, not the other way around. The execd  processes basically "phone home". So you have to run qping  from the exec nodes, not the master node!

Syntax example (I am running this on a exec node, and sheridan is the SGE master):

qping sheridan 536 qmaster 1

where 536  is the port that qmaster  is listening on, and 1  simply means that I am trying to reach a daemon. Can't reach it? Make sure your firewall has a hole on that port, that the routing is correct, that you can ping using the good old ping  command, that the qmaster  process is actually up, and so on.

Of course, you could ping the exec nodes from the master node, too, e.g. I can see if I can reach exec node kosh like this:

 $ qping kosh 537 execd 1

but why would you do such a crazy thing? execd  is responsible for reaching qmaster, not the other way around.

If the above checks out, check the messages log in /var/log/sge_messages on the submit and/or master node (on our Babylon Cluster, they're both the node sheridan ):

 $ tail /var/log/sge_messages

Personally, I like running:

 $ tail -f /var/log/sge_messages

before I submit the job, and then submit a job in a different window. The -f  option will update the tail of the file as it grows, so you can see the message log change "live" as your job executes and see what's happening as things take place.

(Note that the above is actually a symbolic link I put in to the messages log in the qmaster  spool directory, i.e. /opt/sge/default/spool/qmaster/messages .)

One thing that commonly goes wrong is permissions. Make sure that the user that submitted the job using qsub  actually has the permissions to write error, output, and other files to the paths you specified.

For even more precise troubleshooting... maybe the problem is unique only to some nodes(s) or some queue(s)? To pin it down, try to run the job only on some specific node or queue:

 $ qsub -l hostname=<node/host name> <other job params>  

 $ qsub -l qname=<queue name> <other job params> 

Maybe you should also try to SSH into the problem nodes directly and run the job locally from there, as your own user, and see if you can get any more detail on why it fails.

If all else fails...

Sometimes, the SGE master host will become so FUBARed that we have to resort to brute, traumatizing force to fix it. The following solution is equivalent to fixing a wristwatch with a bulldozer, but seems to cause more good than harm (although I can't guarantee that it doesn't cause long-term harm in favor of a short-term solution).

Basically, you wipe the database that keeps track of SGE jobs on the master host, taking any problem "stuck" jobs with it. (At least that's what I think  this does...)

I've found this useful when:

• You submit >10,000 jobs to SGE, which uses too much system resources resulting in their inability to get dispatched to exec hosts, and start getting the "failed receiving gdi request" error on something as simple as qstat. You can't use

  qdel 

  to wipe the jobs due to the same error.
• A job is stuck in the r state (and if you try to delete it, the dr state) despite the fact that the exec host is not running the job, not is even aware of it. This can happen if you reboota stuck/unresponsive exec host.

The solution:

ssh sheridan
su -
service sgemaster stop
cd /opt/sge/default/
mv spooldb spooldb.fubared
mkdir spooldb
cp spooldb.fubared/sge spooldb/
chown -R sgeadmin:sgeadmin spooldb
service sgemaster start

Wipe spooldb.fubared when you are confident that you won't need its contents again.

N1 Grid Engine 6 User's Guide

• Previous: Pending Jobs Not Being Dispatched
• Next: Troubleshooting Common Problems

A job enters the error state when the grid engine system tries to run a job but fails for a reason that is specific to the job.

A queue enters the error state when the grid engine system tries to run a job but fails for a reason that is specific to the queue.

The grid engine system offers a set of possibilities for users and administrators to gather diagnosis information in case of job execution errors. Both the queue and the job error states result from a failed job execution. Therefore the diagnosis possibilities are applicable to both types of error states.

• User abort mail. If jobs are submitted with the qsub -m a command, abort mail is sent to the address specified with the -M user[@host] option. The abort mail contains diagnosis information about job errors. Abort mail is the recommended source of information for users.
• qacct accounting. If no abort mail is available, the user can run the qacct -j command. This command gets information about the job error from the grid engine system's job accounting function.
• Administrator abort mail. An administrator can order administrator mails about job execution problems by specifying an appropriate email address. See under administrator_mail on the sge_conf(5) man page. Administrator mail contains more detailed diagnosis information than user abort mail. Administrator mail is the recommended method in case of frequent job execution errors.
• Messages files. If no administrator mail is available, you should investigate the qmaster messages file first. You can find entries that are related to a certain job by searching for the appropriate job ID. In the default installation, the qmaster messages file is sge-root/cell/spool/qmaster/messages.

  You can sometimes find additional information in the messages of the execd daemon from which the job was started. Use qacct -j job-id to discover the host from which the job was started, and search in sge-root/cell/spool/host/messages for the job ID.

• Previous: Pending Jobs Not Being Dispatched

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019