|Home||Switchboard||Unix Administration||Red Hat||TCP/IP Networks||Neoliberalism||Toxic Managers|
|May the source be with you, but remember the KISS principle ;-)|
|News||Sysadmin Horror Stories||Recommended Links||Creative uses of rm||Mistakes made because of the differences between various Unix/Linux flavors||Missing backup horror stories||Lack of testing complex, potentially destructive, commands before execution of production box||Pure stupidity|
|Locking yourself out||Premature or misguided optimization||Reboot Blunders||Performing the operation on a wrong server||Executing command in a wrong directory||Side effects of performing operations on home or application directories||Typos in the commands with disastrous consequences||Side effects of patching|
|Multiple sysadmin working on the same box||Side effects of patching of the customized server||Ownership changing blunders||Dot-star-errors and regular expressions blunders||Excessive zeal in improving security of the system||Unintended consequences of automatic system maintenance scripts||LVM mishaps||Abuse of privileges|
|Humor Chronicle||Top 10 Classic Unix Humor||Softpanorama Humor Archive||Perl||Shell||Linus Torvalds||Networking||Debugging|
|Safe-rm||Workaholism and Burnout||Coping with the toxic stress in IT environment||The Unix Haterís Handbook||Tips||Horror stories History||Humor||Etc|
|Unix: Modern operating system carefully crafted
to prevent administrators from shooting themselves in the foot.
First of all, rephrasing Oscar Wilde, stupidity is rarely pure and never simple. Reasons for disastrous blunders vary but in many cases they are more like a result of unique confluence of circumstances then purely voluntary blunder of sysadmin.
Sometime trying to meet tight deadline (and being exhausted), or in the situation when "nothing works" ( and being frustrated) people mechanically (on "autopilot") type wrong, but similar command as if part of the brain functions autonomously from the rest ( for example mkfs.ext3 instead of fsck.ext3; or /etc instead of etc) and instantly recognize the blunder, but after hitting Enter key. One simple recommendation is to make hitting Enter key providing a prompt in "fishy" circumstances but writing wrappers for such commands as rm, chown, reboot and like.; Another typical source of blunders is using find -exec option without sufficient testing under time pressure. Hurry slowly is one of the saying that are very true for sysadmin. Sometimes your emotional state contribute to the problems: you didnít have much sleep or your mind was distracted by your personal life problems. In such days it is important to slow down and be extra cautious.
Often some stupid mistake is complicated by the subsequent attempt of cover-up as it is too embarrassing to admit the real cause of the problem. Users and fellow sysadmins are usually good at breaking stuff by ''changing nothing'', ''touching nothing'' or even "doing nothing'"... Commands like rm -rf, chown -r, chmod -r, kill -9, and shutdown/reboot/halt troika often play nasty tricks with people only peripherally acquainted with Unix but for some reason who are given root access.
If I had to hire a Unix sysadmin, the first thing I'd look for is experience. Nothing can substitute for real-life experience in this field and often the way to acquire it are living through some really nasty situations. In Army jargon a bad situation, mistake, or cause of trouble often is called SNAFU. Here is one interesting example:
... My friend was forced to work with some sysadmins who didn't have their act together. One day, one of them was "cleaning" the filesystem and saw a file called "vmunix" in /. "Hmm, this is taking up a lot of space - let's delete it". "rm /vmunix".
My friend had to reinstall the entire OS on that machine after his coworker did this "cleanup". Ahh, the hazards of working with sysadmins who really shouldn't be sysadmins in the first place.
Few more examples from the Unofficial Unix Administration Horror Story Summary:
A friend of mine called me up
saying he no longer could log into his system. I asked him what he had
done recently, and found out that he thought that all executable programs
in /bin, /usr/bin /etc and so on should be owned by
bin, since they were all binaries! So he had chown'ed them
A consultant we had hired (and not a very good one) was installing
Unix on one our workstations. He was mucking with creating and
deleting /dev/tty* files and made /dev/tty a regular file. Weird
things started to happen. Commands would only print their output
if you pressed return twice, etc. Fortunately, we solved the problem
by re-mknod-ing /dev/tty. However, it took a while to realize
what was causing this problem.
I had lost an unknown percentage of the financial data for the institution
that I was working for at the time, right when they were being audited!
Yikes! Anyway, we were able to recover the data and life returned
to normal but I did wonder at the time whether I could still keep my
Moral: make damn sure that *no one* is doing anything on your system before you reboot, even if other users are vociferously clamoring for you to reboot.
Typos in the commands with disastrous consequences are rare, but pressing Enter before checking the command can lead to a real SNAFU:
I once automatically typed /etc instead of etc trying to delete directory to free space on a backup directory on a production server (/etc probably in engraved in sysadmin head as it is typed so often and can be substituted for etc subconsciously). I realized that it was mistake and cancelled the command, but it was a fast server and one third of /etc was gone. The rest of the day was spoiled... Actually not completely: I learned quite a bit about the behavior of AIX in this situation and the structure of AIX /etc directory this day so each such disaster is actually a great learning experience, almost like one day training course ;-). But it's much less nerve wracking to get this knowledge from the course...
Another interesting thing is having backup was not enough is this case -- backup software stopped working. The same was true for telnet and ssh. And this was a remote server is a datacenter across the country. I restored the directory on the other non-production server (overwriting its /etc directory with the help of operations, tell me about cascading errors and Murphy law :-) netcat helped.
In such cases network services with authentication stop working and
the only way to transfer files is using CD/DVD, USB drive or
netcat. That's why it
is useful to have netcat on servers:
netcat is the last resort
file transfer program when services with authentication like ftp
or scp stop working. It is especially useful to have it if the
datacenter is remote.
Just imagine having the sendmail.cf file in /etc. Now, I was working on the sendmail stuff and had come up with lots of sendmail.cf.xxx which I wanted to get rid of so I typed "rm -f sendmail.cf. *"
At first I was surprised about how much time it took to remove some 10 files or so. Hitting the interrupt key, when I finally saw what had happened was way to late, though.
Fortune has it that I'm a very lazy person. That's why I never bothered
to just back up directories with data that changes often. Therefore
I managed to restore /etc successfully before rebooting... :-) Happy
end, after all. Of course I had lost the only well working version of
rm -rf /tmp/foo/bar/ *
rm -rf /tmp/foo/bar/*
Under the category of "learn from the mistake of others..."
About eight years ago, I was working on a program with tight deadlines. I'd worked through the night, only catching an hour or two of sleep in the office.
The next morning, one of the servers remounted it's file systems read-only. Being a small shop, I decided to just take the server down to run a quick fsck.ext2. In my sleepiness though, I typed 'mkfs.ext2'.
When people say that "root" is god, well, no one asks god "Are you sure?".
All nighters are bad news, mistakes are easily made at these times as
we have all learnt the hard way ;)
*cough* erased the backups and spent the night re-backing up data so
nothing actually got done *cough*
I do remember spending a few days putting together some systems check
for my self and my colleague to use such as daily, weekly and monthly
systems checks for all IT aspects (physical, virtual, power,
redundancy, connectivity etc...) only to have something fail the next
day (so it really paid off!) and then nothing has broken since?...Just
goes to show you never know!
Also recently upgraded my personal Ubuntu server to a RAID 6 from a
RAID 5 (about a week ago) and now it looks like one of the drives is
dying, again, just in time!
There are 10 kinds of people in the world; Those who understand
Vigesimal, and 9 others...?
Does this go to show the value of preparedness? Or does it illustrate the
power of luck? Or some intersection? I've often been lucky about when and
how stuff breaks down. And I've known people with what looked like real
computer jinxes. On the one hand, you never want to just trust your luck. On
the other, if luck can be involved, could it be that the profession selects
for those who have it?
I think Whit you have raised some deeper questions maybe about
probability, sod's law, the uncertanty principle, karma, etc
etc...Maybe a venn diagram covering luck and preparedness is in order,
who knows, we/I am digressing....
I would like to point out that at home I'm pretty sure I'm jinxed; my
ubuntu server has decided X ins't going to work any more, nor the
sound (may be related) and the raid is dying, all on the same
There are 10 kinds of people in the world; Those who understand
Vigesimal, and 9 others...?
Way back in the stone age, I was a sys admin at my university, working the graveyard (i.e., backup) shift two days a week and an occasional Sunday. On Sundays, we did the full backup and restore, but we switched out the disk packs (I said this was a long time ago) so we never lost more than a week's worth of data at the time. Well, almost never....
My last Sunday there, I accidentally reinitialized all the disks after the backup but before I had switched them. Then, I realized what I did, switched them anyway, and reinitialized them again, then did a full restore.
Everything would have been fine if the file system hadn't crashed that Friday afternoon....
This was on a Xerox Sigma 7 (I'm dating myself).
UNIX horror story: 24 years ago, I was working on a development system (i.e., nothing critical on it) and my latest build didn't work the way I expected, so I erased it with an 'rm -rf *' - except that I was in the root directory at the time, not my build directory. By the time I realized what I had done, it was too far gone to recover, so I wound up reinstalling the whole system.
No harm done (I did things like that sometimes on purpose, when it was *my* machine involved), but I don't do 'rm -rf' of anything any more without double-checking where I am FIRST, even if the default "-v" is set.
(unsigned confession) ===
I had quite simmilar experience, but I typed `chown -R user:group' /
(instead of ./). Now I'm also checking it for few times and I learned to
use `.' instead of `./', :)
A carpet layer had just finished installing carpet for a lady. He stepped out for a smoke, only to realize he'd lost his cigarettes. He went back in and in the middle of the room, under the carpet, was a bump. "No sense pulling up the entire floor for one pack of smokes," he said to himself. He got out his hammer and flattened the hump.
As he was cleaning up, the lady came in. "Here," she said, handing him his pack of cigarettes. "I found them in the hallway."
"Now," she said, "If only I could find my parakeet."
- Surely the 4 sysadmins of the apocalypse should be:
edquota, rm -rf, kill -9, and shutdown
- What about the four lusers of the apocalypse?
"advertising", "can't log in", "power switch" and "what backup?"
- Sysadmin Olympics :
- 10-base-T Cable Tracing (you'd know what I mean if you saw our site!)
- LUser Avoidance (100 yard race to the bathroom while outmaneuvering LUsers)
- UNIX SUDO wars (four people cuthroat combat..no fair using sudo csh)
- Vendor Poker (try to figure out which vendor is NOT bluffing)
- Find the correct backup tape (I know it's amongst these unlabeled DAT tapes in my desk)
- System disk recovery (Without a valid backup on a disk with a headcrash, see above)
- Write system configurations manual, without technical words.
- Guess what the Luser is really typing (see the csh> cd tilda or $set def sysdollarsystem or csh> VI DOTRHOSTS)
- When I first started working with sendmail, I was convinced that the cf file had been created by someone bashing their head on the keyboard. After a week, I realised this was, indeed, almost certainly the case
Modern operating system carefully crafted to prevent administrators from shooting themselves in the foot.
 Interestingly, most utilities have a command line option which will cause the system to rip the user's legs off and beat them to death with the soggy ends. This is often the default behaviour.
- kill -9 them all, let reboot -rf now sort them out
- I've found that things like "If you change even one configuration setting and your system ceases to function, or functions in a manner other than expected, our support staff will laugh at you in the sinister manner of Joseph Stalin just before he enslaved eastern Europe" helps to draw peoples attention to essential details like this.
- The Strong Lusethropic Principle states: "The more idiot proof the software, the more it encourages the user to be careless and not think. Therefore, idiot-proof software actually encourages, contributes, and actually CAUSES lusers to be stupid."
The Weak Lusethropic Principle states: "As more idiot-proof software becomes avalable, more idiots are able to use computers. Idiot-proof software did not make or cause computer lusers; it simple allowed lusers to use computers where they could not before."
- Sysadmins don't go to hell; we're already doing our time in purgatory.
- I think I'd like to see a Simpsons episode start up with Bart Simpson writing 'I will not attempt to undermine the Usenet cabal'.
- <fantasy mode>
"Sir, Sir! I've deleted all my files"
"Yes my child, and pray tell me how"
"I wanted to delete directories fred1 and fred2 but I typed rm -rf fred * when should have typed rm -rf fred*"
"Ah ha, and what have you learnt"
"To beware of powerful file name globbing facilities that my shell provides for my careful use."
"Yes my child, you have learnt a painful lession. Now you must learn how to recover files from one of the many, multiply redundent backup tapes you have carefully written every evening ever since you were granted a powerful personal workstation."
"Master, I'm eager to learn!"
- Can you SysAdmins tell me what might go on in a typical day?
Hours of endless frustration punctuated by moments of sheer terror.
- life suddenly made much more sense, the day i fully grokked that people are stupid.
- To sysadmin or not to sysadmin... that is the question, whether tis nobler in the minde to suffer the slings and arrowes of outragious fortune, or climb to the top of the building with a fucking high-power rifle and scope.
Greg "Twotone" Spiegelberg
- I think that we should officially make this the sysadmins credo. We'll call it "The Abigail Oath" and require all new sysadmins to swear it.
Well, without the layoff part, maybe something like this:
I am hired because I know what I am doing, not because I will do whatever I am told is a good idea. This might cost me bonuses, raises, promotions, and may even label me as "undesirable" by places I don't want to work at anyway, but I don't care. I will not compromise my own principles and judgement without putting up a fight. Of course, I won't always win, and I will sometimes be forced to do things I don't agree with, but if I am my objections will be known, and if I am shown to be right and problems later develop, I will shout "I told you so!" repeatedly, laugh hysterically, and do a small dance or jig as appropriate to my heritage.
- Same to you, dipshit -- Coredump
- I start to be a wandering sysadmin on the 10th Jan. -- John Burnham
Do you get your own bard as well?
"Bravely bold Sir Burnham
Brought forth from Camelot.
He was not afraid to die,
Oh, brave Sir Burnham!
He was not at all afraid to be killed in nasty ways.
Brave, brave, brave Sir Burnham.
He was not in the least bit scared to be mashed into a pulp.
Or to have his cables gouged out, and his MX records broken!
To have his domains split, and his /dev burned away
And his file systems all hacked and mangled, brave Sir Burnham.
His newsrc smashed in and his heart cut out,
And his relays removed and his routers unplugged,
And his hubs baked and his soul burnt off,
And his peni--"
- Fsck, either way I'm screwed. -- petro
Now *that* is the Sysadmin's motto. -- Peter da Silva
- I used to herd dairy cows. Now I herd lusers. Apart from the isolation, I think I preferred the cows. They were better conversation, easier to milk, and if they annoyed me enough, I could shoot them and eat them.
- Sigh. Anyone want an irritable, manic-depressive sysadmin ? -- John
I'll think about it. Do you think I should keep a backup for the one I already have? -- Bernard Peek
No. You run them both in a cluster environment, so they can share the irritability. Of course, having two sysadmin's you naturally get twice the irritability than before. If one sysadmin goes down then the other one becomes twice as irritable to cope.
Benefits in this environment allow one sysadmin to go offline for maintenance (eg beer!) without major impact. Unfortunately most clusters of this type are poorly configured and one sysadmin going offline generally causes the other to go offline as well.
- Damn, squid must have satisfied my reloads from cache. -- Peter da Silva
That sounds so very much nastier than it is. -- adam
- Eh? Linux is luserproof? What kind of "proper" set up is that, ripping out all removable media devices and ethernet, freezing the hard drive spindle, encasing it in concrete and dropping it off a pier?
- One day, a student asked a master, "Master, there is conflict between the suits and the sysadmins. Which group has the Zen nature, and which group is grieviously disturbing the stillness of the Tao?"
And the master said nothing, but installed an operating system. And the student was enlightened.
- A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.
- A disturbance of sysadmins. Because if we are, you will be.
- A Zen of SysAdmins
We are at one with our work. If you disturb our work, our foot will be at one with your ass. Sadly, you won't be at One with anything thereafter. You'll be in lots of little pieces we call Bits, some of which are at One with themselves, and others, which are at Zero with themselves, but none of which will be At One with you.
Redirection is hard." -- Web-Luser Barbie -- flaps
- In every given certification course at least one student must decide that each person in the class came n-thousand miles to hear his/her own questionable "expertise" delivered unprompted and continued ad nauseum much to the dismay of the rest of the class.
- I may be a Bastard, but I never ever ever argue with flight crew. They just don't have a sense of humor about that sort of thing.
If the stewardess tells me NT is a far more stable and reliable OS than UNIX, I'll nod politely and say "Is it really? How nice."
Of course, I'll rant about what a luser she is later, but as long as we're on the plane, she gets to be uid zero.
Choose Windows. Choose a PC. Choose a fucking big hard disk and a Pentium III to run Internet Explorer and mIrc. Choose HTML email, and viruses... Melissa, Wm.Concept, ExploreZip. Choose sitting in that chair watching mind-numbing soulless web-pages, stuffing fucking virus-infected plugins into your browser. Choose porn, rotting away at the end of it all, pishing away your last on a miserable dialup, staring at fat whores, nothing but an embarassment to the people who built the net.
Doesn't matter what a luser chooses, they're still a luser.
Peter da Silva
Remember to chant "Pie Jesu domine, dona eis requiem" whenever you do that.
-- Peter Gutmann
I can't do that, Peter. I have several old texts in my office (bound with animal skin, and inked with blood), and every time I attempt to chant any sort of Latin the walls begin to bleed, and my Arrowhead Mountain Spring Water bottle tells me it's going to kill me with my flathead screwdriver.
-- Stephen S. Edwards II
- If you tell them, they never listen. If they listen, they never learn. If they learn, they never remember. If they remember, they never obey.
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least
Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info|
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: July 01, 2018