Performing the operation on a wrong server

"You are trapped in a maze of screens and ssh sessions all alike."
"It is dark, and you are likely to log off the wrong account."

Nep.

The typical case of the loss of situational awareness is performing some critical operation on the wrong server.  There are two useful rules here

1. If you use Windows desktop to connect to Unix servers ALWAYS change background for each in your terminal emulator. For example if you work with two servers and has tqo sessions on for each use shade of green background for one and shade of blue background for another. That might help to avoid nasty blunders connected with executing destructive command (often deletion of the file) on the wrong server. 
2. If you prefer to work as root, then switch to root only on one server at a time and exit when you are finished. Use your regular ID and sudo on the all others.

Such commands as reboot or mkinitrd can be pretty devastating when applied to wrong server. Especially in this is an important production server. This situation often is made more probable due to not fault-tolerant name scheme employed in many corporations where names of the servers differ by just one symbol. For example, scheme serv01, serv02, serv03 and so on is a pretty dangerous name scheme as server names are different by only single digit and thus errors like working on the wrong server are much more probable.  You can use mnemonic name of servers in you prompt instead of standard host names by adding include that specify mnemonic name of the server in /etc directory. Something like /etc/hostalias  and display if instead or along with "real" hostname in your prompt.

Even more complex scheme like bsn01dls9, nyc02per6 were first three letter encode the  location, then numeric suffix and then vendor of the hardware and OS installed  are prone to such errors. My impression that unless first letters differ, there is a substantial chance of  working on wrong server. Using mnemonic names for example, your favorite sport teams names is a better strategy and those names can be defined as aliases both in /etc/hosts and in DNS (if you have some level of control on DNS). 

The servers that you accidentally use might have very similar name , for example XYZ300 instead of XYZ200 or their functions are very similar (production and quality boxes). that's why reboot command generally should have wrapper which asks whether you are trying to reboot wrong server. As reboot command often is used under very stressful situation, alias to the wrapper is very important safety measure.   As aliases are not used in non-interactive session it does not affect your scripts (but using reboot in scripts is also not a very good idea, you should provide some delay and use shutdown instead.  If the time argument is used, 5 minutes before the system goes down the /run/nologin file is created to ensure no further logins.

In any case showing the mnemonic  name of the host and different color of the background in your terminal emulator are two more or less effective effective measures against  performing operation of the wrong server. That's why it's important that shell prompt shows not only hostname, but also the mnemonic  name of the host. Which implies that shell promote should consist of two lines. The first line is "status" line  and the second should sho the minimal information (rc code and user name) in order not to clutter the line on which you type the commands. For example

function my_prompt
{
   local EXIT_STATUS=$?
   color_yellow="\[\e[33;40m\]"
   color_red_bold="\[\e[31;1m\]"
   color_green_bold="\[\e[32;1m\]"
   color_blue_bold="\[\e[34;1m\]"
   color_none="\[\e[0m\]"
   echo `date +"%y/%m/%d %H:%M"` "$PWD ============================$HOSTNAME $HOSTALIAS"
   local ps1_status ps1_user_color

   if (( $EXIT_STATUS != 0 )); then
      ps1_status="${color_yellow}[$EXIT_STATUS]${color_none} "
   else
      ps1_status="[0]"
   fi

   if [[ `whoami` = "root" ]]; then
      ps1_user_color=${color_red_bold}
   else
      ps1_user_color=${color_blue_bold}
   fi

   local name=`hostname -s`
   PS1="${ps1_status}${ps1_user_color}\\u@$name:${color_none} \\$ "
}

Often, if you both have production computer and quality server for some application is wise never have two terminals opened simultaneously as root. Either change one to  a regular user or move it to a separate workspace. Closing and reopening when it needed is also not a big deal, but  can save you from some very unpleasant situations.

For example here are some education examples of this situation (from IT Resource Center forums - greatest blunders)

Michael Steele

When I was first starting out I worked for a Telecom as an 'Application Administrator' and I sat in a small room with a half a dozen other admins and together we took calls from users as their calls escalated up from tier I support. We were tier II in a three tier organization.

A month earlier someone from tier I confused a production server with a test server and rebooted it in the middle of the day. These servers were remotely connected over a large distance so it can be confusing. Care is needed before rebooting.

The tier I culprit took a great deal of abuse for this mistake and soon became a victim of several jokes. An outage had been caused in a high availability environment which meant management, interviews, reports; It went on and on and was pretty brutal.

And I was just as brutal as anyone. Their entire organization soon became victimize by everyone from our organization. The abuse traveled right up the management tree and all participated. It was hilarious, for us.

Until I did the same thing a month later. There is nothing more humbling then 2000 people all knowing who you are for the wrong reason and I have never longed for anonymity more.

Now I always do a 'uname' or 'hostname' before a reboot, even when I'm right in front of it.

Paula J Frazer-Campbell

Hi
I spent half a day trying to trace a fault on a machine -- could not find anything wrong - strange!!!

Went back to start, to discover I was working on the wrong server.

Paula

YLTan

We have two cdrom drive mounted side by side. Both are use by different server. In a mid of rush, I put in a CD in the wrong drive and try to mount that CD in another using "mount /dev/dsk/c1t2d0 /CDROM" and it keep saying "Device Busy".

PANIC!!! thinking my server have broken, I called HP for help, only to discover the CD is in the wrong drive....how embarrassing.!!

Paula J Frazer-Campbell

Scene

Two K class server side by side logged into backup server telnet to live to check a setting forgot to exit and wished to reboot backup :-

shutdown -r now

Many cries from outside of the server room as the live system with 650+ users trundled around on a 6 way 8 gig K class reboot - this is measured in days not mins.

Tim Adamson

I had a HP Engineer on site to install a heap of hardware in a machine. I shutdown and halted the server, then proceeded to power off the entire cabinet. Pity I was 4 feet out and powered off production cabinet.

My excuse - damned lowsy power, we really should have a UPS :p

NEWS CONTENTS

• 20181005 : I corrupted a 400TB data warehouse. ( Oct 05, 2018 , www.reddit.com )

Old News ;-)

[Oct 05, 2018] I corrupted a 400TB data warehouse.

Oct 05, 2018 | www.reddit.com

I corrupted a 400TB data warehouse.

Took 6 days to restore from tape.

• permalink

mcowger VCDX | DevOps Guy 8 points 9 points 10 points 5 years ago (0 children)

Meh - happened a long time ago.

Had a big Solaris box (E6900) running Oracle 10 for the DW. Was going to add some new LUNs to the box and also change some of the fiber pathing to go through a new set of faster switches. Had the MDS changes prebuilt, confirmed in with another admin, through change control, etc.

Did fabric A, which went through fine, and then did fabric B without pausing or checking that the new paths came up on side A before I knocked over side B (in violation of my own approved plan). For the briefest of instants, there were no paths to the devices and Oracle was configured in full async write mode :(. Instant corruption of the tables that were active. Tried to do use archivelogs to bring it back, but no dice (and this is before Flashbacks, etc). So we were hosed.

Had to have my DBA babysit the RMAN restore for the entire weekend :(. 1GBe links to backup infrastructure.

RCA resulted in MANY MANY changes to the design of that system, and me just barely keeping my job.

invisibo DevOps 2 points 3 points 4 points 5 years ago (0 children)

You just made me say "holy shit! Out loud. You win.

FooHentai 2 points 3 points 4 points 5 years ago (0 children)

Ouch.

I dropped a 500Gb RAID set. There were 2 identical servers in the rack right next to each other. Both OpenFiler, both unlabeled. Didn't know about the other one and was told to 'wipe the OpenFiler'. Got a call half an hour later from a team wondering where all their test VMs had gone.

vocatus NSA/DOD/USAR/USAP/AEXP [ S ] 1 point 2 points 3 points 5 years ago (0 children)

I have to hear the story.

Recommended Links

Softpanorama Recommended

Top articles

Sites

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: January 28, 2019