Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

Pure stupidity

A collection of educational and horrifying tales of SGS (Spontaneous Gross Stupidity)

News Sysadmin Horror Stories Recommended Links An observation about corporate security departments Mistakes made because of the differences between various Unix/Linux flavors Missing backup horror stories Lack of testing complex, potentially destructive, commands before execution of production box Creative uses of rm
Locking yourself out Premature or misguided optimization Reboot Blunders  Performing the operation on a wrong server Executing command in a wrong directory Side effects of performing operations on home or application directories Typos in the commands with disastrous consequences Side effects of patching
Multiple sysadmin working on the same box Side effects of patching of the customized server Ownership changing blunders Dot-star-errors and regular expressions blunders Excessive zeal in improving security of the system Unintended consequences of automatic system maintenance scripts LVM mishaps Abuse of privileges
Humor Chronicle Top 10 Classic Unix Humor Softpanorama Humor Archive Perl Shell Linus Torvalds  Networking Debugging
Safe-rm Workaholism and Burnout Coping with the toxic stress in IT environment The Unix Haterís Handbook Tips Horror stories History Humor Etc
  Unix: Modern operating system carefully crafted to prevent administrators from shooting themselves in the foot[1].

[1] Interestingly, most utilities have a command line option which will cause the system to rip the user's legs off and beat them to death with the soggy ends. This is often the default behaviour.

Bruce Murphy

The consequences of any action are never be fully understood until after it's too late to do anything about it.

Eric The Read

If all you have is an axe, every problem looks like hours of fun.


"Unix for Dummies" is surely a single page pull out with "don't" printed on it.


First of all, rephrasing Oscar Wilde, stupidity is rarely pure and never simple. Reasons for disastrous blunders vary but in many cases they are more like a result of unique confluence of circumstances then purely voluntary blunder of sysadmin. Often the most stupid actions are done as a reaction to the problem and desire to fix it quickly without analyzing all the available information (for example if NFS server hangs and you can do nothing with it)

The most common example is attempt to reboot the box without gathering all the nessesary information about the problems (and all evidence will be destoryed by the reboot, which might or might not help).  A good role to follow is that "emergency reboot", if possible, can be done only after one hour of analyzing the situation.

Problems with the server  often cascade and for example hang NFS server exposes "embellishments" for root dot files sources form NFS filesystem. And you can't login to root without cancelling  dot files by execution with Ctrl-C. But to understand that there is a problem with your dot files in addition to problem with the NFS server and they are related requires some time.  In this sense using the default dot files of the root user is probably a good practice.

As Linux is excessively complex OS. That means that from one problem to another you usually forget most of the vital information, or the fact that there were similar situations in the past, so instead of jumping into action reading your notes about previous crash often helps. This is probably the most intelligent  step in troubleshooting you can do. Assuming you have such notes. Not having them is another example of poor stupidity ;-).

Sometime trying to meet tight deadline  (and being exhausted), or in the situation when "nothing works" ( and being frustrated) people mechanically (on "autopilot") type wrong, but similar command as if part of the brain functions autonomously from the rest ( for example mkfs.ext3 instead of fsck.ext3; or /etc instead of etc) and instantly recognize the blunder, but after hitting Enter key. One  simple recommendation is to make hitting Enter key providing a prompt in "fishy" circumstances but writing wrappers for such commands as rm, chown, reboot and like.; Another typical source of blunders is using find -exec option without sufficient testing under time pressure. Hurry slowly is one of the saying that are very true for sysadmin. Sometimes your emotional state contribute to the problems: you didnít have much sleep or your mind was distracted by your personal life problems. In such days it is important to slow down and be extra cautious.

Often some stupid mistake is complicated by the subsequent attempt of  cover-up as it is too embarrassing to admit the real cause of the problem.  Users and fellow sysadmins are usually good at breaking stuff by ''changing nothing'', ''touching nothing'' or even "doing nothing'"...  Commands like rm -rf, chown -r, chmod -r, kill -9, and shutdown/reboot/halt  troika often play nasty tricks with people only peripherally acquainted with Unix but for some reason who are given root access. 

If I had to hire a Unix sysadmin, the first thing I'd look for is experience.  Nothing can substitute for real-life experience in this field and often the way to acquire it are living through some really nasty situations. In Army jargon  a bad situation, mistake, or cause of trouble often is called SNAFU.  Here is one interesting example:

... My friend was forced to work with some sysadmins who didn't have their act together.  One day, one of them was "cleaning" the filesystem and saw a file called "vmunix" in /. "Hmm, this is taking up a lot of space - let's delete it".  "rm /vmunix".

My friend had to reinstall the entire OS on that machine after his coworker did this "cleanup".  Ahh, the hazards of working with sysadmins who really shouldn't be sysadmins in the first place.

Few more examples from the Unofficial Unix Administration Horror Story Summary:

Typos in the commands with disastrous consequences are rare, but pressing Enter before checking the command can lead to a real SNAFU:

A classic, probably unmatched case of corporate security departments stupidity: blocking ICMP

  TL;DR, short for "too long; didn't read", is Internet slang to say that some text being replied to has been ignored because of its length. In slang it can also stand for "Too lazy; didn't read". It is also used as a signifier for a summary of an online post or news article.

[Jan 10, 2019]  When idiots are offloaded to security department, interesting things with network eventually happen

Security department often does more damage to the network then any sophisticated hacker can.  Especially if they are populated with morons, as they usually are.  One of the  most blatant examples is below... Those idiots decided to disable Traceroute (which means ICMP) in order to increase security.
An this is not a unique case; I observed such cases in large companies.
May 27, 2018 |

jfdavis668 ( 1414919 ) , Sunday May 27, 2018 @11:09AM ( #56682996 )

Re:So ( Score: 5 , Interesting)

Traceroute is disabled on every network I work with to prevent intruders from determining the network structure. Real pain in the neck, but one of those things we face to secure systems.

Anonymous Coward writes:
Re: ( Score: 2 , Insightful)

What is the point? If an intruder is already there couldn't they just upload their own binary?

Hylandr ( 813770 ) , Sunday May 27, 2018 @05:57PM ( #56685274 )
Re: So ( Score: 5 , Interesting)

They can easily. And often time will compile their own tools, versions of Apache, etc..

At best it slows down incident response and resolution while doing nothing to prevent discovery of their networks. If you only use Vlans to segregate your architecture you're boned.

gweihir ( 88907 ) , Sunday May 27, 2018 @12:19PM ( #56683422 )
Re: So ( Score: 5 , Interesting)

Also really stupid. A competent attacker (and only those manage it into your network, right?) is not even slowed down by things like this.

bferrell ( 253291 ) , Sunday May 27, 2018 @12:20PM ( #56683430 ) Homepage Journal
Re: So ( Score: 4 , Interesting)

Except it DOESN'T secure anything, simply renders things a little more obscure... Since when is obscurity security?

fluffernutter ( 1411889 ) writes:
Re: ( Score: 3 )

Doing something to make things more difficult for a hacker is better than doing nothing to make things more difficult for a hacker. Unless you're lazy, as many of these things should be done as possible.

DamnOregonian ( 963763 ) , Sunday May 27, 2018 @04:37PM ( #56684878 )
Re:So ( Score: 5 , Insightful)


Things like this don't slow down "hackers" with even a modicum of network knowledge inside of a functioning network. What they do slow down is your ability to troubleshoot network problems.

Breaking into a network is a slow process. Slow and precise. Trying to fix problems is a fast reactionary process. Who do you really think you're hurting? Yes another example of how ignorant opinions can become common sense.

mSparks43 ( 757109 ) writes:
Re: So ( Score: 2 )

Pretty much my reaction. like WTF? OTON, redhat flavors all still on glibc2 starting to become a regular p.i.t.a. so the chances of this actually becoming a thing to be concerned about seem very low.

Kinda like gdpr, same kind of groupthink that anyone actually cares or concerns themselves with policy these days.

ruir ( 2709173 ) writes:
Re: ( Score: 3 )

Disable all ICMP is not feasible as you will be disabling MTU negotiation and destination unreachable messages. You are essentially breaking the TCP/IP protocol. And if you want the protocol working OK, then people can do traceroute via HTTP messages or ICMP echo and reply.

Or they can do reverse traceroute at least until the border edge of your firewall via an external site.

DamnOregonian ( 963763 ) , Sunday May 27, 2018 @04:32PM ( #56684858 )
Re:So ( Score: 4 , Insightful)

You have no fucking idea what you're talking about. I run a multi-regional network with over 130 peers. Nobody "disables ICMP". IP breaks without it. Some folks, generally the dimmer of us, will disable echo responses or TTL expiration notices thinking it is somehow secure (and they are very fucking wrong) but nobody blocks all ICMP, except for very very dim witted humans, and only on endpoint nodes.

DamnOregonian ( 963763 ) writes:
Re: ( Score: 3 )

That's hilarious... I am *the guy* who runs the network. I am our senior network engineer. Every line in every router -- mine.

You have no idea what you're talking about, at any level. "disabled ICMP" - state statement alone requires such ignorance to make that I'm not sure why I'm even replying to ignorant ass.

DamnOregonian ( 963763 ) writes:
Re: ( Score: 3 )

Nonsense. I conceded that morons may actually go through the work to totally break their PMTUD, IP error signaling channels, and make their nodes "invisible"

I understand "networking" at a level I'm pretty sure you only have a foggy understanding of. I write applications that require layer-2 packet building all the way up to layer-4.

In short, he's a moron. I have reason to suspect you might be, too.

DamnOregonian ( 963763 ) writes:
Re: ( Score: 3 )

A CDS is MAC. Turning off ICMP toward people who aren't allowed to access your node/network is understandable. They can't get anything else though, why bother supporting the IP control channel? CDS does *not* say turn off ICMP globally. I deal with CDS, SSAE16 SOC 2, and PCI compliance daily. If your CDS solution only operates with a layer-4 ACL, it's a pretty simple model, or You're Doing It Wrong (TM)

nyet ( 19118 ) writes:
Re: ( Score: 3 )

> I'm not a network person

IOW, nothing you say about networking should be taken seriously.

kevmeister ( 979231 ) , Sunday May 27, 2018 @05:47PM ( #56685234 ) Homepage
Re:So ( Score: 4 , Insightful)

No, TCP/IP is not working fine. It's broken and is costing you performance and $$$. But it is not evident because TCP/IP is very good about dealing with broken networks, like yours.

The problem is that doing this requires things like packet fragmentation which greatly increases router CPU load and reduces the maximum PPS of your network as well s resulting in dropped packets requiring re-transmission and may also result in widow collapse fallowed with slow-start, though rapid recovery mitigates much of this, it's still not free.

It's another example of security by stupidity which seldom provides security, but always buys added cost.

Hylandr ( 813770 ) writes:
Re: ( Score: 3 )

As a server engineer I am experiencing this with our network team right now.

Do you have some reading that I might be able to further educate myself? I would like to be able to prove to the directors why disabling ICMP on the network may be the cause of our issues.

Zaelath ( 2588189 ) , Sunday May 27, 2018 @07:51PM ( #56685758 )
Re:So ( Score: 4 , Informative)

A brief read suggests this is a good resource: []

Bing Tsher E ( 943915 ) , Sunday May 27, 2018 @01:22PM ( #56683792 ) Journal
Re: Denying ICMP echo @ server/workstation level t ( Score: 5 , Insightful)

Linux has one of the few IP stacks that isn't derived from the BSD stack, which in the industry is considered the reference design. Instead for linux, a new stack with it's own bugs and peculiarities was cobbled up.

Reference designs are a good thing to promote interoperability. As far as TCP/IP is concerned, linux is the biggest and ugliest stepchild. A theme that fits well into this whole discussion topic, actually.


Top Visited
Past week
Past month


Old News ;-)

[Jul 08, 2010] OT SysAdmin Stories

Under the category of "learn from the mistake of others..."

About eight years ago, I was working on a program with tight deadlines. I'd worked through the night, only catching an hour or two of sleep in the office.

The next morning, one of the servers remounted it's file systems read-only. Being a small shop, I decided to just take the server down to run a quick fsck.ext2. In my sleepiness though, I typed 'mkfs.ext2'.

When people say that "root" is god, well, no one asks god "Are you sure?".


All nighters are bad news, mistakes are easily made at these times as we have all learnt the hard way ;)

*cough* erased the backups and spent the night re-backing up data so nothing actually got done *cough*

I do remember spending a few days putting together some systems check for my self and my colleague to use such as daily, weekly and monthly systems checks for all IT aspects (physical, virtual, power, redundancy, connectivity etc...) only to have something fail the next day (so it really paid off!) and then nothing has broken since?...Just goes to show you never know!

Also recently upgraded my personal Ubuntu server to a RAID 6 from a RAID 5 (about a week ago) and now it looks like one of the drives is dying, again, just in time!


There are 10 kinds of people in the world; Those who understand Vigesimal, and 9 others...?


Does this go to show the value of preparedness? Or does it illustrate the power of luck? Or some intersection? I've often been lucky about when and how stuff breaks down. And I've known people with what looked like real computer jinxes. On the one hand, you never want to just trust your luck. On the other, if luck can be involved, could it be that the profession selects for those who have it?



I think Whit you have raised some deeper questions maybe about probability, sod's law, the uncertanty principle, karma, etc etc...Maybe a venn diagram covering luck and preparedness is in order, who knows, we/I am digressing....

I would like to point out that at home I'm pretty sure I'm jinxed; my ubuntu server has decided X ins't going to work any more, nor the sound (may be related) and the raid is dying, all on the same day?!?!?!


There are 10 kinds of people in the world; Those who understand
Vigesimal, and 9 others...?


Way back in the stone age, I was a sys admin at my university, working the graveyard (i.e., backup) shift two days a week and an occasional Sunday. On Sundays, we did the full backup and restore, but we switched out the disk packs (I said this was a long time ago) so we never lost more than a week's worth of data at the time. Well, almost never....

My last Sunday there, I accidentally reinitialized all the disks after the backup but before I had switched them. Then, I realized what I did, switched them anyway, and reinitialized them again, then did a full restore.

Everything would have been fine if the file system hadn't crashed that Friday afternoon....

This was on a Xerox Sigma 7 (I'm dating myself).

UNIX horror story: 24 years ago, I was working on a development system (i.e., nothing critical on it) and my latest build didn't work the way I expected, so I erased it with an 'rm -rf *' - except that I was in the root directory at the time, not my build directory. By the time I realized what I had done, it was too far gone to recover, so I wound up reinstalling the whole system.

No harm done (I did things like that sometimes on purpose, when it was *my* machine involved), but I don't do 'rm -rf' of anything any more without double-checking where I am FIRST, even if the default "-v" is set.

(unsigned confession) ===

I had quite simmilar experience, but I typed `chown -R user:group' /
(instead of ./). Now I'm also checking it for few times and I learned to
use `.' instead of `./', :)

Dominik Zyla

[Jun 26, 2010] Non-sysadmin related but pretty amusing story about some wrong assumptions ;-)

A carpet layer had just finished installing carpet for a lady. He stepped out for a smoke, only to realize he'd lost his cigarettes. He went back in and in the middle of the room, under the carpet, was a bump. "No sense pulling up the entire floor for one pack of smokes," he said to himself. He got out his hammer and flattened the hump.

As he was cleaning up, the lady came in. "Here," she said, handing him his pack of cigarettes. "I found them in the hallway."

"Now," she said, "If only I could find my parakeet."

[Jun 21, 2010] Memorable Quotes from Alt.Sysadmin.Recovery


Top Visited
Past week
Past month

Recommended Links

Google matched content

Softpanorama Recommended

Top articles




Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy


War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes


Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law


Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2018 by Dr. Nikolai Bezroukov. was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case is down you can use the at


The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 07, 2019