Softpanorama May the source be with you, but remember the KISS principle ;-)	Home	Switchboard	Unix Administration	Red Hat	TCP/IP Networks	Neoliberalism	Toxic Managers
	(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix

Mass permissions and ownership changing blunders

News	Sysadmin Horror Stories	Recommended Links	Creative uses of rm	Mistakes made because of the differences between various Unix/Linux flavors	Missing backup horror stories	Lack of testing complex, potentially destructive, commands before execution of production box	Pure stupidity
Locking yourself out	Premature or misguided optimization	Reboot Blunders	Performing the operation on a wrong server	Executing command in a wrong directory	Side effects of performing operations on home or application directories	Typos in the commands with disastrous consequences	Side effects of patching
Multiple sysadmin working on the same box	Side effects of patching of the customized server	Ownership changing blunders	Dot-star-errors and regular expressions blunders	Excessive zeal in improving security of the system	Unintended consequences of automatic system maintenance scripts	LVM mishaps	Abuse of privileges
Safe-rm	Workaholism and Burnout	Coping with the toxic stress in IT environment	The Unix Hater’s Handbook	Tips	Horror stories History	Humor	Etc

Those are classic blunders which are committed at the beginning of their carrier by almost any sysadmins. Please remember that before using chown with find you need to test the command first using ls or similar option.

From: [email protected] (Jerry Rocteur)
Organization: InCC.com Perwez Belgium
Horror story,

I sent one of my support guys to do an Oracle update in Madrid.

As instructed he created a new user called esf and changed the files
in /u/appl to owner esf, however in doing so he *must* have cocked up
his find command, the command was:
```
 find /u/appl -user appl -exec chown esf {} \;
```
He rang me up to tell me there was a problem, I logged in via x25 and
about 75% of files on system belonged to owner esf.

VERY little worked on system.

What a mess, it took me a while and I came up with a brain wave to fix it but it really screwed up the system.

Moral: be *very* careful of find execs, get the syntax right!!!!
From: [email protected] (Marc Fraioli)
Organization: Grebyn Timesharing
Well, here's a good one for you:

I was happily churning along developing something on a Sun workstation, and was getting a number of annoying permission denieds from trying to write into a directory hierarchy that I didn't own. Getting tired of that, I decided to set the permissions on that subtree to 777 while I was working, so I wouldn't have to worry about it. Someone had recently told me that rather than using plain "su", it was good to use "su -", but the implications had not yet sunk in. (You can probably see where this is going already, but I'll go to the bitter end.) Anyway, I cd'd to where I wanted to be, the top of my subtree, and did su -. Then I did chmod -R 777. I then started to wonder why it was taking so damn long when there were only about 45 files in 20 directories under where I (thought) I was. Well, needless to say, su - simulates a real login, and had put me into root's home directory, /, so I was proceeding to set file permissions for the whole system to wide open. I aborted it before it finished, realizing that something was wrong, but this took quite a while to straighten out.

Marc Fraioli
- Oh you bastards. I was hoping that a thread like this would never appear, because if it did, I knew I would have to confess. Oh well...
  About a year back, I was looking through /etc and found that a few system files had world write permission. Gasping with horror, I went to put it right with something like
  
  dipshit# chmod -r 664 /etc/*
  
  (I know, I know, goddamnit!.. now)
  
  Everything was OK for about two to three weeks, then the machine went down for some reason (other than the obvious). Well, I expect that you can imagine the result. The booting procedure was unable to run fsck, so barfed and mounted the file systems read-only, and bunged me into single-user mode. Dumb expression..gradual realisation..cold sweat.
  
  Of course, now I can't do a frigging chmod +x on anything because it's all read-only. In fact I can't run anything that isn't part of sh.
  
  Wedgerama. Hysteria time. Consider reformatting disks. All sorts of crap ideas. Headless chicken scene. Confession.
  
  "You did WHAT??!!"
  
  Much forehead slapping, solemn oaths and floor pacing.
  
  Luckily, we have a local MegaUnixGenius who, having sat puzzled for an hour or more, decided to boot from a cdrom and take things from there. He fixed it.
  
  My boss, totally amazed at the fix I'd got the system into, luckilysaw the funny side of it. I didn't. Even though at that stage, I didn't know much about unix/suns/booting/admin, I did actually know enough to NOT use a command like the one above. Don't ask. Must be the drugs.
  
  BTW, if my future employer _is_ reading this (like they say he/she might), then I have certainly learned tonnes of stuff in the last year, especially having had to set up a complete Sun system, fix local problems, etc :-)
  
  Anyone else got a tale of SGS (Spontaneous Gross Stupidity) ?
  
  -dave "I'm much better now, honest.. no, really.. hey what's this button doooooooooOOOOOO..."

Top Visited <p>Your browser does not support iframes.</p>					Switchboard
					Latest
					Past week
					Past month

NEWS CONTENTS

20181005 : Unix Admin. Horror Story Summary, version 1.0 by Anatoly Ivasyuk ( Oct 05, 2018 , cam.ac.uk )
20181005 : One wrong find command can create one weak frantic recovery efforts ( Oct 05, 2018 , cam.ac.uk )
20180422 : Unix-Linux Horror Stories Unix Horror Stories The good thing about Unix, is when it screws up, it does so very quickly ( Aug 04, 2011 , unixhorrorstories.blogspot.com )
20100708 : OT SysAdmin Stories ( OT SysAdmin Stories, Jul 08, 2010 )

Old News ;-)

[Oct 05, 2018] Unix Admin. Horror Story Summary, version 1.0 by Anatoly Ivasyuk

Oct 05, 2018 | cam.ac.uk

From: [email protected] (Marc Fraioli)
Organization: Grebyn Timesharing

Well, here's a good one for you:

I was happily churning along developing something on a Sun workstation, and was getting a number of annoying permission denieds from trying to
write into a directory heirarchy that I didn't own. Getting tired of that, I decided to set the permissions on that subtree to 777 while Iwas working, so I wouldn't have to worry about it.

Someone had recently told me that rather than using plain "su", it was good to use "su -", but the implications had not yet sunk in. (You can probably see where this is going already, but I'll go to the bitter end.)

Anyway, I cd'd to where I wanted to be, the top of my subtree, and did su -. Then I did chmod -R 777. I then started to wonder why it was taking so damn long when there were only about 45 files in 20 directories under where I (thought) I was. Well, needless to say, su - simulates a real login, and had put me into root's home directory, /, so I was proceeding to set file permissions for the whole system to wide open.

I aborted it before it finished, realizing that something was wrong, but this took quite a while to straighten out.

Marc Fraioli

[Oct 05, 2018] One wrong find command can create one weak frantic recovery efforts

This is a classic SNAFU known and described for more then 30 years. It is still repeated in various forms thousand time by different system administrators. You can get permission of file installed via RPM back rather quickly and without problems. For all other files you need a backup or educated guess.

Ahh, the hazards of working with sysadmins who are not ready to be sysadmins in the first place

Oct 05, 2018 | cam.ac.uk

From: [email protected] (Jerry Rocteur)
Organization: InCC.com Perwez Belgium

Horror story,

I sent one of my support guys to do an Oracle update in Madrid.

As instructed he created a new user called esf and changed the files
in /u/appl to owner esf, however in doing so he *must* have cocked up
his find command, the command was:

find /u/appl -user appl -exec chown esf {} \;

He rang me up to tell me there was a problem, I logged in via x25 and
about 75% of files on system belonged to owner esf.

VERY little worked on system.

What a mess, it took me a while and I came up with a brain wave to
fix it but it really screwed up the system.

Moral: be *very* careful of find execs, get the syntax right!!!!

[Apr 22, 2018] Unix-Linux Horror Stories Unix Horror Stories The good thing about Unix, is when it screws up, it does so very quickly

Notable quotes:

"... And then I realized I had thrashed the server. Completely. ..."

"... There must be a way to fix this , I thought. HP-UX has a package installer like any modern Linux/Unix distribution, that is swinstall . That utility has a repair command, swrepair . ..."

"... you probably don't want that user owning /bin/nologin. ..."

Aug 04, 2011 | unixhorrorstories.blogspot.com

Unix Horror Stories: The good thing about Unix, is when it screws up, it does so very quickly The project to deploy a new, multi-million-dollar commercial system on two big, brand-new HP-UX servers at a brewing company that shall not be named, had been running on time and within budgets for several months. Just a few steps remained, among them, the migration of users from the old servers to the new ones.

The task was going to be simple: just copy the home directories of each user from the old server to the new ones, and a simple script to change the owner so as to make sure that each home directory was owned by the correct user. The script went something like this:
#!/bin/bash
cat /etc/passwd|while read line
      do
         USER=$(echo $line|cut -d: -f1)
         HOME=$(echo $line|cut -d: -f6)
         chown -R $USER $HOME
      done
[NOTE: the script does not filter out system ids from userids and that's a grave mistake. also it was run before it was tested ; -) -- NNB]

As you see, this script is pretty simple: obtain the user and the home directory from the password file, and then execute the chown command recursively on the home directory. I copied the files, executed the script, and thought, great, just 10 minutes and all is done.

That's when the calls started.

It turns out that while I was executing those seemingly harmless commands, the server was under acceptance test. You see, we were just one week away from going live and the final touches were everything that was required. So the users in the brewing company started testing if everything they needed was working like in the old servers. And suddenly, the users noticed that their system was malfunctioning and started making furious phone calls to my boss and then my boss started to call me.

And then I realized I had thrashed the server. Completely. My console was still open and I could see that the processes started failing, one by one, reporting very strange messages to the console, that didn't look any good. I started to panic. My workmate Ayelen and I (who just copied my script and executed it in the mirror server) realized only too late that the home directory of the root user was / -the root filesystem- so we changed the owner of every single file in the filesystem to root!!! That's what I love about Unix: when it screws up, it does so very quickly, and thoroughly.

There must be a way to fix this , I thought. HP-UX has a package installer like any modern Linux/Unix distribution, that is swinstall . That utility has a repair command, swrepair . So the following command put the system files back in order, needing a few permission changes on the application directories that weren't installed with the package manager:

swrepair -F

But the story doesn't end here. The next week, we were going live, and I knew that the migration of the users would be for real this time, not just a test. My boss and I were going to the brewing company, and he receives a phone call. Then he turns to me and asks me, "What was the command that you used last week?". I told him and I noticed that he was dictating it very carefully. When we arrived, we saw why: before the final deployment, a Unix administrator from the company did the same mistake I did, but this time, people from the whole country were connecting to the system, and he received phone calls from a lot of angry users. Luckily, the mistake could be fixed, and we all, young and old, went back to reading the HP-UX manual. Those things can come handy sometimes!

Morale of this story: before doing something on the users directories, take the time to see which is the User ID of actual users - which start usually in 500 but it's configuration-dependent - because system users's IDs are lower than that.

Send in your Unix horror story, and it will be featured here in the blog!

Greetings,
Agustin

Colin McD, 16 de marzo de 2017, 15:02

This script is so dangerous. You are giving home directories to say the apache user and you probably don't want that user owning /bin/nologin.

[Jul 08, 2010] OT SysAdmin Stories

UNIX horror story: 24 years ago, I was working on a development system (i.e., nothing critical on it) and my latest build didn't work the way I expected, so I erased it with an 'rm -rf *' - except that I was in the root directory at the time, not my build directory.

By the time I realized what I had done, it was too far gone to recover, so I wound up reinstalling the whole system. No harm done (I did things like that sometimes on purpose, when it was *my* machine involved), but I don't do 'rm -rf' of anything any more without double-checking where I am FIRST, even if the default "-v" is set.
(unsigned confession)

I had quite similar experience, but I typed `chown -R user:group' / (instead of ./). Now I'm also checking it for few times and I learned to use `.' instead of `./', :)
--
Dominik Zyla

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: October 26, 2018

Etc