Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers May the source be with you, but remember the KISS principle ;-)

# Creative uses of rm with unintended consequences

### Wiping out much more then you intended to do

 News Enterprise Unix System Administration Recommended Links Simple Unix Backup Tools Unix rm command Unix mv command Missing backup horror stories Mistakes made because of the differences between various Unix/Linux flavors Creative uses of rm Lack of testing complex, potentially destructive, commands before execution of production box Pure stupidity Performing the operation on a wrong server Safe-rm Typical Errors In Using Find Tips Unix History Humor Etc

### Introduction

Many Unix sysadmin horror stories are related to unintended consequences such as side effects of classic Unix commands. One of the most prominent of such commands is rm.   Which has several idiosyncrasies which are iether easily forgotten or unknown by novice sysadmins. For example behavior of rm -r .* can easily be forgotten from one encounter to another, especially if sysadmin works both with Unix and Windows servers. Consequences if executed as root are tragic...

While most, probably 99.9% of operations with rm are successful it is the remaining.01% that create a lot of damage and is subject to sysadmin folklore.

The command rm has relatively few options. Linux uses GNU implementation that as of April 2018 has the following options (note relatively new option -I):

-f, --force
ignore nonexistent files, never prompt
-i
prompt before every removal
-I
prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes
--interactive[=WHEN]
prompt according to WHEN: never, once (-I), or always (-i). Without WHEN, prompt always
--one-file-system
when removing a hierarchy recursively, skip any directory that is on a file system different from that of the corresponding command line argument
--no-preserve-root
do not treat '/' specially
--preserve-root
do not remove '/' (default)
-r, -R, --recursive
remove directories and their contents recursively
-v, --verbose
explain what is being done
--help
display this help and exit
--version
output version information and exit

Wiping out useful data by using rm with wrong parameters or options is probably  the most common blunder by both novice and seasoned sysadmins. It happens rarely but the result are often devastating.  This danger can't be completely avoided, but in addition to having an up-to-date backup, there are several steps that you can do to reduce the chances of the major damage to the data:

1. Try to use option -I (instead of -i). While -i option (which it default alas in Red Hat and other distributions) is annoying and most often user disable it, there is more modern and more useful option -I which is highly recommended. Unfortunately very few sysadmin know about its existence. It prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes (rm(1) remove files-directories ):

If the -I or --interactive=once option is given, and there are more than three files or the -r, -R, or --recursive are given, then rm prompts the user for whether to proceed with the entire operation. If the response is not affirmative, the entire command is aborted.

2. Block operation for system level 2 directories such  as /etc or /usr (this simple measure actually is now implemented in GNU rm used in Linux) using wrapper such  as safe-rm. Aliasing rm to wrapper is safe because aliases work only in interactive sessions. In addition you can block the deletion of directories that are in your favorite directories list if you have any.  for example if you use level 2 directory /Fav for links some often used directories, you can check is the directory you are trying to delete is present in those links.
3. Rename the directory before applying rm -r command to a special name DELETEIT or some other mnemonic  name and then deleting it using special alias dirdel or something like that (this way you will never mistype the name of the directory): many devastating blunders with rm occur when you operate on backup copy of the  system or important user/application directory. And you automatically type the name of the directory with slash in front of it  instread of proper relative name (rm -r /etc instead of rm -r etc ) because you are conditioned to type it this way by a long practive and this sequnce of keystroke is "wired" in your brain and you do it almost semi-consciously. If you first rename the directory to something else, for example DELETEIT first,  such an error will not occur . A script can check if there is another directory with this name in existence and warn you too.  You can also simply move directory to /Trash folder which has, say 90 day expiration period for files in it
4. Use a protective script as an envelope to rm. It can  that take into account some of the most dangerous situation typical in your environment. You can write a script like rmm or saferm  which among other useful preventive checks introduces a delay between hitting enter and  starting of the operation and list the file or at lest number of them and first five that will be affected.

### Classic blunders with RM command

There are several classic blunders committed using the rm command :

1. Running rm - R command without testing it using ls . You should always use ls -Rl command to test complex rm -R  commands (  -R, --recursive means process  subdirectories recursively).
2. Mistyping complex path or file name in rm commands. It's always better to use copy and paste operation for directories and files used in rm command as it helps to avoid various typos. If you do recursive deletion is useful to have a "dry run" using find or ls to see the files.

Here are to example of typos:

rm -rf /tmp/foo/bar/ *

rm -rf /tmp/foo/bar/*

===

rm -r /etc

rm -r etc

This actually is an interesting type of error because /etc is typed daily so often that it kind of ingrained in sysadmin head and can be typed subconsciously instead of etc

3. Using .*   That essentially make rm recursive as it matches ".." (parent directory).
4. You can accidentally hit Enter by mistake before finishing typing the line containing rm command.

Those cases can be viewed as shortcomings of rm implementation (it should automatically block * deletion of system directories like //etc/ and so on unless -f  flag is specified. As well as any system file.  Also Unix does not have system attribute for files although sticky bit on files can be used instead along with ownership of sys instead of root).

### Writing a wrapper like "saferm"

Unix lacks system attributes for files although sticky bit on files or immutable attribute can be used instead.  It is wise to use wrappers for rm. There are several more or less usable approaches for writing such a wrapper:

• Alias rm to a Perl script with configurable blacklist of files and directories that should never be removed. This is an approach implemented in Perl script safe-rm
• Redefining rm as mv to , say, /var/junk directory which should be cleaned periodically with find -mtime -7 or greater.
• Displaying several first targets before executing actual rm command. In this case rm can be wrapped with the shell function or script, as in command line rm is usually typed without path.
• Some combination of those approaches

In view of  danger of  rm -r unleashed as root it is wise to use wrappers for rm and alias rm to this wrapper in root .bash_profile or other profile file that you are using for root.  Aliases are not used in non-interactive sessions. So it will not affect any scripts. And in command line sessions rm is usually typed without path (which creates another set of dangers).

There are several more or less usable features that might wish to experiment with when writing such a wrapper:

• Configurable blacklist of files and directories that should never be removed) This is an approach implemented in Perl script safe-rm
• Redefining rm as mv to junk directory which should be cleaned periodically with find -mtime -7 or greater.
• Using an ls of find command that displays several first targets before executing actual rm command.
• Calculate number of files and total size that you intend to delete and introduce delay for confirmation  of your action.
• If this is a directory that exists somewhere else (can be verified via locate command) rename it to unique name before the deletion
• Do not allow more then one argument if -r option is used.
• Introduce a sleep interval of, say, 5 second -- enough to realize the consciences of your action

Of course each environment is unique and such a wrapper should take into account idiosyncrasies of such an environment.

If you know Perl, you can use Safe-rm as the initial implementation and enhance it. If it licensed under GPL. It present in Ubuntu I think.

How to use
-----------

Once you have installed safe-rm on your system (see INSTALL), you will need to
fill the system-wide or user-specific blacklists with the paths that you'd like
to protect against accidental deletion.

The system-wide blacklist lives in /etc/safe-rm.conf and you should probably add
paths like these:

/
/etc
/usr
/usr/lib
/var

The user-specific blacklist lives in ~/.config/safe-rm and could include things like:

Other approaches
-----------------

If you want more protection than what safe-rm can offer, here are a few suggestions.

You could of course request confirmation every time you delete a file by putting this in

alias rm='rm -i'

But this won't protect you from getting used to always saying yes, or from accidently
using 'rm -rf'.

Or you could make use of the Linux filesystem "immutable" attribute by marking (as root)
each file you want to protect:

chattr +i file

Of course this is only usable on filesystems which support this feature.

Here are two projects which allow you to recover recently deleted files by trapping

delsafe
http://homepage.esoterica.pt/~nx0yew/delsafe/

libtrashcan

There are also projects which implement the FreeDesktop.org trashcan spec. For example:

trash-cli

Finally, this project is a fork of GNU coreutils and adds features similar to safe-rm
to the rm command directly:

http://wiki.github.com/d5h/rmfd/

### Backup files before executing rm command, if you deleting a directory and it is not that large

Backups before execution of rm command are important. For example making backup of /etc directory on modern server takes a couple of seconds, but can save quite a lot of nerves in situations that otherwise can s can be devastating. For example, in the example above you would erase all your files and subdirectories in the /etc directory. Modern flavors of Unix usually prevent erasing / but not /etc. Linux which uses GNU version of rm prevents erasing all level 2 system directories.

You can also move directories to /Trash folder and delete them after, say 7 days or so.

You can also move the directory to /Trash folder is it is relatively small. In this forlder you can delete files that are say, older then 7 days with a cron script, if the total size of this directory exceeds certain threshold.

### More on blunders with the deleting a directory from the backup

I once automatically typed /etc instead of etc trying to delete directory to free space on a backup directory on a production server (/etc probably in engraved in sysadmin head as it is typed so often and can be substituted for etc subconsciously).  I realized that it was mistake and cancelled the command, but it was a fast server and one third of /etc was gone. The rest of the day was spoiled...  Actually not completely: I learned quite a bit about the behavior of AIX in this situation and the structure of AIX /etc directory this day so each such disaster is actually a great learning experience, almost like one day training course ;-). But it's much less nerve wracking to get this knowledge from the course... Another interesting thing is having backup was not enough is this case -- enterprise backup software stopped working on a damaged server. The same was true for telnet and ssh. And this was a remote server is a datacenter across the country.  I restored the directory on the other non-production server (overwriting its /etc directory in this second box with the help of operations, tell me about cascading errors and Murphy law :-). Then netcat helped to transfer the tar file.

 If you are working as root and perform dangerous operations never type a path of the command, copy it from the screen. If you can copy command from history instead of typing, do it !

Such blunders are really persistent as often used directories are types in a "semiconscious" fashion, in "autopilot" mode and you realize the blunder only after you hit Enter. For example, many years after the first blunder, I similarly typed rm -r /Soft instead of rm -r Soft in a backup directory.  And I was in a hurry so I did not even bother to load my profile and word on "bare root" account which did not have this "saferm" feature I was talking about earlier. Unfortunately for me /Soft was a huge directory so backup was not practical as I needed to free space.  It with all kind of software in it and the server was very fast to in a couple of seconds that elapsed before I realized the blunder approximately half gigabyte of files and directories was wiped out.  Luckily I have previous day backup.

In such cases network services with authentication stop working and the only way to transfer files is using CD/DVD, USB drive or netcat. That's why it is useful to have netcat on servers: netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

 netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-).

 The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-). Please read classic sysadmin horror stories.

A simple extra space often produced horrible consequences:

cd /usr/lib
ls /tmp/foo/bar

I typed
rm -rf /tmp/foo/bar/ *

rm -rf /tmp/foo/bar/*

The system doesn’t run very will without all of it’s libraries……

You can block such behavior requiring that if -r option was given you should have one and only one argument to rm. That should be a part of functionality of your "saferm" script.

### Important class of subtle Unix errors: dot-star errors

Another poplar class of recursive rm errors are so called dot-star-errors, which often happn when rm is used with find.  Novice sysadmins usually do not realize that '.*' also matches '..' often with disastrous consequences.  If you are in any doubt as for how a wildcard will expand use echo command to see the result.

• From: robjohn@ocdis01.UUCP (Contractor Bob Johnson)
Organization: Tinker Air Force Base, Oklahoma

Another horror story (mine this time)...

Cleaning out an old directory, I did 'rm *', then noticed several files that began with dot (.profile, etc) still there.  So, in a fit of obtuse brilliance, I typed...

rm -rf .* &

By the time I got it stopped, it had chewed through 3 filesystems which all had to be restored from tape (.* expands to ../*, and the -r makes it keep walking up the directory tree).  Live and learn...

• My mistake on SunOS (with OpenWindows) was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed

"rm -rf ./.*".

I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first.

### Using "convenience" symlinks to other (typically deeply nested) directories inside a directories and forgetting about them

Command  rm follows symlinks. So if you have a symlink to some system on important directory from the directory you are deleting all files in them will be deleted.

That's the main danger of "convenience symlinks" which are used to simplify access to deeply nested directories.  They are more dangerous then useful. Using aliases or functions is a better deal. If you use them, do this only from the level two directory created specially for this purpose like /Fav and protect this directory in your saferm script.

### Disasterious typos in name of the directory or regex -- unintended space

There are some exotic typos that can lean you to troubles, especially with -r  option. One of them is unintended space:

 rm /homejoeuser/old*

rm /homejoeuser/old*

Or, similarly:

 rm * _old

rm *old
In all cases when you are deleting the directory directly copying its name from the listing obtained via ls  or find command is much safer that typing it by yourself.

Such a mistake is more damaging  is you use -r option. For example:

rm -rf /tmp/foo/bar/ *

rm -rf /tmp/foo/bar/*

So it is prudent to block execution of rm commands with multiple argument, if option -r is used in your saferm script.

### Root deletion protection

To remove a file you must have write permission on the folder where it is stored. Sun introduced "rm -rf /" protection in Solaris 10, first released in 2005. Upon executing the command, the system now reports that the removal of / is not allowed.

Shortly after, the same functionality was introduced into FreeBSD version of rm  utility.

GNU rm  refuses to execute rm -rf /  if the --preserve-root  option is given, which has been the default since version 6.4 of GNU Core Utilities was released in 2006.

No such protection exists for critical system directories like /etc /bin, and so on but you can imitate it putting file "-i" into such directories or using a wrapper for the interactive usage of the command.  See also Safe-rm

### Using echo command to see expansion of your wildcards, if any

To understand what will be actually being executed after shell expansion, preface your rm command with echo. For example if there is a filename starting with the dash you will receive very confusing message from rm

	$rm * rm: unknown option -- - usage: rm [-f|-i] [-dPRrvW] file ... To find out what caused it prefix the command with echo echo rm * You can generate list of files dynamically for rm (piping it via xarg if it is too large. For example ls | grep copy # check what files will be affected ls | grep copy | xargs -0 -l rm -f # remove all the files The simplest attempt to defend themselves against accidentally deleting files is to create an alias rm or function rm along the lines of: alias rm="rm -i" or use the function function rm { /bin/rm -i "$@" ; }

Unfortunately, this encourage a tendency to mindlessly pound y  and the return key to affirm removes - until you realize that got just past the file you need to keep. Therefore it is better to create a function which first displays the files and then delete them:

function rm
{
if [ -z "$PS1" ] ; then /bin/rm "$@"
exit $? fi ls -l "$@"
echo 'remove[ny]? ' | tr -d '\012' ;
if [ "_$REPLY" = "_y" ]; then /bin/rm "$@"
exit $? fi echo '(rm command cancelled)' exit 1 }  If this function is included into startup dot files it will be found ahead of the system binary /bin/rm in the search path and can break sloppy written batch jobs. That's why it checks for PS1 being non empty but this is not a 100% reliable test. ### Using file starting with - as stoppers for unintended use of rm. Undeletable files The rm command accepts the -- option which will cause it to stop processing flag options from that point forward. This allows the removal of file names that begin with a dash (-). rm -- -filename Actually accidental creation of files starting with dash is a common problem for novice sysadmin. The problem is that if I type rm -foo, the rm command treats the filename as an option. There are two simple ways to delete such a file. One was shown above and based on supplying an empty option argument. The second to use a relative or absolute pathname: rm ./-badfile  rm /home/user/-badfile  Here the debugging trick that we mentioned above -- prefacing rm command with echo really helps to understand what happens. To delete a file with non-printable characters in the name you can also use the shell wildcard "?" for each bad character rm bad?file?name Another way to delete files that have control characters in the name is to use -i option and an asterisk, which gives you the option of removing each file in the directory — even the ones that you can't type. % rm -i * rm: remove faq.html (y/n)? n rm: remove foo (y/n)? y % The -i option may also be helpful when you are dealing with files with Unicode characters that appear to be regular letters in your locale, but that don't match patterns or names you use otherwise. A great way to discover files with control characters in them is to use the -q option to the Unix ls command (some systems also support a useful -b option). You can, for example, alias the ls command: alias lsq='ls -q ' Files that have control characters in their filenames will then appear with question marks: ls -q f* faq.html fo?o ### Blunders in using rm with find This is class or blunders that deserves a page of its own. See Typical Errors In Using Find You need to be extra careful running rm in an -exec option of find command. There are many horror stories in which people delete large parts of their filesystem due to typos of sloppy find command parameters. Again it is prudent first to test it using ls command to see the actual list of files to be deleted. Sometimes it is quite different from what you intended to do :-). I often first generate the list using ls command, inspect it and in a separate step pipe it into rm : cat rm.lst | xargs -t -n1 rm  The -t flag causes the xargs command to display each command before running it, so you can see what is happening. Dr. Nikolai Bezroukov  Top Visited Your browser does not support iframes. Switchboard Latest Past week Past month ## NEWS CONTENTS ## Old News ;-) #### [Apr 22, 2018] THE classic Unix horror story programming ##### Looks like not much changed since 1986. I amazed how little changed with Unix over the years. RM remains a danger although zsh and -I option on gnu rm are improvement. . I think every sysadmin wiped out important data with rm at least once in his career. So more work on this problem is needed. ##### Notable quotes: ##### "... Because we are creatures of habit. If you ALWAYS have to type 'yes' for every single deletion, it will become habitual, and you will start doing it without conscious thought. ..." ##### "... Amazing what kind of damage you can recover from given enough motivation. ..." ##### "... " in "rm -rf ~/ ..." ###### Apr 22, 2008 | www.reddit.com probablycorey 10 years ago (35 children) A little trick I use to ensure I never delete the root or home dir... Put a file called -i in / and ~ If you ever call rm -rf *, -i (the request confirmation option) will be the first path expanded. So your command becomes... rm -rf -i Catastrophe Averted! mshade 10 years ago (0 children) That's a pretty good trick! Unfortunately it doesn't work if you specify the path of course, but will keep you from doing it with a PWD of ~ or /. Thanks! aythun 10 years ago (2 children) Or just use zsh. It's awesome in every possible way. brian@xenon:~/tmp/test% rm -rf * zsh: sure you want to delete all the files in /home/brian/tmp/test [yn]?  rex5249 10 years ago (1 child) I keep an daily clone of my laptop and I usually do some backups in the middle of the day, so if I lose a disk it isn't a big deal other than the time wasted copying files. MyrddinE 10 years ago (1 child) Because we are creatures of habit. If you ALWAYS have to type 'yes' for every single deletion, it will become habitual, and you will start doing it without conscious thought. Warnings must only pop up when there is actual danger, or you will become acclimated to, and cease to see, the warning. This is exactly the problem with Windows Vista, and why so many people harbor such ill-will towards its 'security' system. zakk 10 years ago (3 children) and if I want to delete that file?!? ;-) alanpost 10 years ago (0 children) I use the same trick, so either of:$ rm -- -i

or

$rm ./-i will work. emag 10 years ago (0 children) rm /-i ~/-i nasorenga 10 years ago * (2 children) The part that made me the most nostalgic was his email address: mcvax!ukc!man.cs.ux!miw Gee whiz, those were the days... (Edit: typo) floweryleatherboy 10 years ago (6 children) One of my engineering managers wiped out an important server with rm -rf. Later it turned out he had a giant stock of kiddy porn on company servers. monstermunch 10 years ago (16 children) Whenever I use rm -rf, I always make sure to type the full path name in (never just use *) and put the -rf at the end, not after the rm. This means you don't have to worry about hitting "enter" in the middle of typing the path name (it won't delete the directory because the -rf is at the end) and you don't have to worry as much about data deletion from accidentally copy/pasting the command somewhere with middle click or if you redo the command while looking in your bash history. Hmm, couldn't you alias "rm -rf" to mv the directory/files to a temp directory to be on the safe side? branston 10 years ago (8 children) Aliasing 'rm' is fairly common practice in some circles. It can have its own set of problems however (filling up partitions, running out of inodes...) amnezia 10 years ago (5 children) you could alias it with a script that prevents rm -rf * being run in certain directories. jemminger 10 years ago (4 children) you could also alias it to 'ls' :) derefr 10 years ago * (1 child) One could write a daemon that lets the oldest files in that directory be "garbage collected" when those conditions are approaching. I think this is, in a roundabout way, how Windows' shadow copy works. branston 10 years ago (0 children) Could do. Think we might be walking into the over-complexity trap however. The only time I've ever had an rm related disaster was when accidentally repeating an rm that was already in my command buffer. I looked at trying to exclude patterns from the command history but csh doesn't seem to support doing that so I gave up. A decent solution just occurred to me when the underlying file system supports snapshots (UFS2 for example). Just snap the fs on which the to-be-deleted items are on prior to the delete. That needs barely any IO to do and you can set the snapshots to expire after 10 minutes. Hmm... Might look at implementing that.. mbm 10 years ago (0 children) Most of the original UNIX tools took the arguments in strict order, requiring that the options came first; you can even see this on some modern *BSD systems. shadowsurge 10 years ago (1 child) I just always format the command with ls first just to make sure everything is in working order. Then my neurosis kicks in and I do it again... and a couple more times just to make sure nothing bad happens. Jonathan_the_Nerd 10 years ago (0 children) If you're unsure about your wildcards, you can use echo to see exactly how the shell will expand your arguments. splidge 10 years ago (0 children) A better trick IMO is to use ls on the directory first.. then when you are sure that's what you meant type rm -rf !$ to delete it.
earthboundkid 10 years ago * (0 children)
Ever since I got burned by letting my pinky slip on the enter key years ago, I've been typing echo path first, then going back and adding the rm after the fact.
zerokey 10 years ago * (2 children)
Great story. Halfway through reading, I had a major wtf moment. I wasn't surprised by the use of a VAX, as my old department just retired their last VAX a year ago. The whole time, I'm thinking, "hello..mount the tape hardware on another system and, worst case scenario, boot from a live cd!"

Then I got to, "The next idea was to write a program to make a device descriptor for the tape deck" and looked back at the title and realized that it was from 1986 and realized, "oh..oh yeah...that's pretty fucked."

iluvatar 10 years ago (0 children)

Great story

Yeah, but really, he had way too much of a working system to qualify for true geek godhood. That title belongs to Al Viro . Even though I've read it several times, I'm still in awe every time I see that story...

cdesignproponentsist 10 years ago (0 children)
FreeBSD has backup statically-linked copies of essential system recovery tools in /rescue, just in case you toast /bin, /sbin, /lib, ld-elf.so.1, etc.

It won't protect against a rm -rf / though (and is not intended to), although you could chflags -R schg /rescue to make them immune to rm -rf.

clytle374 10 years ago * (9 children)
It happens, I tried a few months back to rm -rf bin to delete a directory and did a rm -rf /bin instead.

First thought: That took a long time.

I was amazed that the desktop survived for nearly an hour before crashing.

earthboundkid 10 years ago (8 children)
This really is a situation where GUIs are better than CLIs. There's nothing like the visual confirmation of seeing what you're obliterating to set your heart into the pit of your stomach.
jib 10 years ago (0 children)
If you're using a GUI, you probably already have that. If you're using a command line, use mv instead of rm.

In general, if you want the computer to do something, tell it what you want it to do, rather than telling it to do something you don't want and then complaining when it does what you say.

earthboundkid 10 years ago (3 children)
Yes, but trash cans aren't manly enough for vi and emacs users to take seriously. If it made sense and kept you from shooting yourself in the foot, it wouldn't be in the Unix tradition.
earthboundkid 10 years ago (1 child)
1. Are you so low on disk space that it's important for your trash can to be empty at all times?
2. Why should we humans have to adapt our directory names to route around the busted-ass-ness of our tools? The tools should be made to work with capital letters and spaces. Or better, use a GUI for deleting so that you don't have to worry about OMG, I forgot to put a slash in front of my space!

Seriously, I use the command line multiple times every day, but there are some tasks for which it is just not well suited compared to a GUI, and (bizarrely considering it's one thing the CLI is most used for) one of them is moving around and deleting files.

easytiger 10 years ago (0 children)
Thats a very simple bash/ksh/python/etc script.
1. script a move op to a hidden dir on the /current/ partition.
2. alias this to rm
3. wrap rm as an alias to delete the contents of the hidden folder with confirmation
mattucf 10 years ago (3 children)
I'd like to think that most systems these days don't have / set as root's home directory, but I've seen a few that do. :/
dsfox 10 years ago (0 children)
This is a good approach in 1986. Today I would just pop in a bootable CDROM.
fjhqjv 10 years ago * (5 children)
That's why I always keep stringent file permissions and never act as the root user.

I'd have to try to rm -rf, get a permission denied error, then retype sudo rm -rf and then type in my password to ever have a mistake like that happen.

But I'm not a systems administrator, so maybe it's not the same thing.

toast_and_oj 10 years ago (2 children)
I aliased "rm -rf" to "omnomnom" and got myself into the habit of using that. I can barely type "omnomnom" when I really want to, let alone when I'm not really paying attention. It's saved one of my projects once already.
shen 10 years ago (0 children)
I've aliased "rm -rf" to "rmrf". Maybe I'm just a sucker for punishment.

I haven't been bit by it yet, the defining word being yet.

robreim 10 years ago (0 children)
I would have thought tab completion would have made omnomnom potentially easier to type than rm -rf (since the -rf part needs to be typed explicitly)
immure 10 years ago (0 children)
It's not.
lespea 10 years ago (0 children)
before I ever do something like that I make sure I don't have permissions so I get an error, then I press up, home, and type sudo <space> <enter> and it works as expected :)
kirun 10 years ago (0 children)
And I was pleased the other day how easy it was to fix the system after I accidentally removed kdm, konqueror and kdesktop... but these guys are hardcore.
austin_k 10 years ago (0 children)
I actually started to feel sick reading that. I've been in a IT disaster before where we almost lost a huge database. Ugh.. I still have nightmares.
umilmi81 10 years ago (4 children)
Task number 1 with a UNIX system. Alias rm to rm -i. Call the explicit path when you want to avoid the -i (ie: /bin/rm -f). Nobody is too cool to skip this basic protection.
flinchn 10 years ago (0 children)
i did an application install at an LE agency last fall - stupid me mv ./etc ./etcbk <> mv /etc /etcbk

ahh that damned period

DrunkenAsshole 10 years ago (0 children)
Were the "*"s really needed for a story that has plagued, at one point or another, all OS users?
xelfer 10 years ago (0 children)
Is the home directory for root / for some unix systems? i thought 'cd' then 'rm -rf *' would have deleted whatever's in his home directory (or whatever $HOME points to) srparish 10 years ago (0 children) Couldn't he just have used the editor to create the etc files he wanted, and used cpio as root to copy that over as an /etc? sRp stox 10 years ago (1 child) Been there, done that. Have the soiled underwear to prove it. Amazing what kind of damage you can recover from given enough motivation. sheepskin 10 years ago * (0 children) I had a customer do this, he killed it about the same time. I told him he was screwed and I'd charge him a bunch of money to take down his server, rebuild it from a working one and put it back up. But the customer happened to have a root ftp session up, and was able to upload what he needed to bring the system back. by the time he was done I rebooted it to make sure it was cool and it booted all the way back up. Of course I've also had a lot of customer that have done it, and they where screwed, and I got to charge them a bunch of money. jemminger 10 years ago (0 children) pfft. that's why lusers don't get root access. supersan 10 years ago (2 children) i had the same thing happened to me once.. my c:\ drive was running ntfs and i accidently deleted the "ntldr" system file in the c:\ root (because the name didn't figure much).. then later, i couldn't even boot in the safe mode! and my bootable disk didn't recognize the c:\ drive because it was ntfs!! so sadly, i had to reinstall everything :( wasted a whole day over it.. b100dian 10 years ago (0 children) Yes, but that's a single file. I suppose anyone can write hex into mbr to copy ntldr from a samba share! bobcat 10 years ago (0 children) http://en.wikipedia.org/wiki/Emergency_Repair_Disk boredzo 10 years ago (0 children) Neither one is the original source. The original source is Usenet, and I can't find it with Google Groups. So either of these webpages is as good as the other. docgnome 10 years ago (0 children) In 1986? On a VAX? MarlonBain 10 years ago (0 children) This classic article from Mario Wolczko first appeared on Usenet in 1986 . amoore 10 years ago (0 children) I got sidetracked trying to figure out why the fictional antagonist would type the extra "/ " in "rm -rf ~/ ". Zombine 10 years ago (2 children) ...it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Yeah. So apart from the fact that no one could get any work done or really do anything, things were working great! I think a more rational reaction would be "Why on Earth is this big, important system on which many people rely designed in such a way that a simple easy-to-make human error can screw it up so comprehensively?" or perhaps "Why on Earth don't we have a proper backup system?" daniels220 10 years ago (1 child) The problem wasn't the backup system, it was the restore system, which relied on the machine having a "copy" command. Perfectly reasonable assumption that happened not to be true. Zombine 10 years ago * (0 children) Neither backup nor restoration serves any purpose in isolation. Most people would group those operations together under the heading "backup;" certainly you win only a semantic victory by doing otherwise. Their fail-safe data-protection system, call it what you will, turned out not to work, and had to be re-engineered on-the-fly. I generally figure that the assumptions I make that turn out to be entirely wrong were not "perfectly reasonable" assumptions in the first place. Call me a traditionalist. #### [Apr 22, 2018] rm and Its Dangers (Unix Power Tools, 3rd Edition) ###### Apr 22, 2018 | docstore.mik.ua 14.3. rm and Its Dangers Under Unix, you use the rm command to delete files. The command is simple enough; you just type rm followed by a list of files. If anything, rm is too simple. It's easy to delete more than you want, and once something is gone, it's permanently gone. There are a few hacks that make rm somewhat safer, and we'll get to those momentarily. But first, here's a quick look at some of the dangers. To understand why it's impossible to reclaim deleted files, you need to know a bit about how the Unix filesystem works. The system contains a "free list," which is a list of disk blocks that aren't used. When you delete a file, its directory entry (which gives it its name) is removed. If there are no more links ( Section 10.3 ) to the file (i.e., if the file only had one name), its inode ( Section 14.2 ) is added to the list of free inodes, and its datablocks are added to the free list. Well, why can't you get the file back from the free list? After all, there are DOS utilities that can reclaim deleted files by doing something similar. Remember, though, Unix is a multitasking operating system. Even if you think your system is a single-user system, there are a lot of things going on "behind your back": daemons are writing to log files, handling network connections, processing electronic mail, and so on. You could theoretically reclaim a file if you could "freeze" the filesystem the instant your file was deleted -- but that's not possible. With Unix, everything is always active. By the time you realize you made a mistake, your file's data blocks may well have been reused for something else. When you're deleting files, it's important to use wildcards carefully. Simple typing errors can have disastrous consequences. Let's say you want to delete all your object ( .o ) files. You want to type: % rm *.o But because of a nervous twitch, you add an extra space and type: % rm * .o It looks right, and you might not even notice the error. But before you know it, all the files in the current directory will be gone, irretrievably. If you don't think this can happen to you, here's something that actually did happen to me. At one point, when I was a relatively new Unix user, I was working on my company's business plan. The executives thought, so as to be "secure," that they'd set a business plan's permissions so you had to be root ( Section 1.18 ) to modify it. (A mistake in its own right, but that's another story.) I was using a terminal I wasn't familiar with and accidentally created a bunch of files with four control characters at the beginning of their name. To get rid of these, I typed (as root ): # rm ????* This command took a long time to execute. When about two-thirds of the directory was gone, I realized (with horror) what was happening: I was deleting all files with four or more characters in the filename. The story got worse. They hadn't made a backup in about five months. (By the way, this article should give you plenty of reasons for making regular backups ( Section 38.3 ).) By the time I had restored the files I had deleted (a several-hour process in itself; this was on an ancient version of Unix with a horrible backup utility) and checked (by hand) all the files against our printed copy of the business plan, I had resolved to be very careful with my rm commands. [Some shells have safeguards that work against Mike's first disastrous example -- but not the second one. Automatic safeguards like these can become a crutch, though . . . when you use another shell temporarily and don't have them, or when you type an expression like Mike's very destructive second example. I agree with his simple advice: check your rm commands carefully! -- JP ] -- ML #### [Apr 22, 2018] How to prevent a mistaken rm -rf for specific folders? ##### Notable quotes: ##### "... There's nothing more on a traditional Linux, but you can set Apparmor/SELinux/ rules that prevent rm from accessing certain directories. ..." ##### "... Probably your best bet with it would be to alias rm -ri into something memorable like kill_it_with_fire . This way whenever you feel like removing something, go ahead and kill it with fire. ..." ###### Jan 20, 2013 | unix.stackexchange.com I think pretty much people here mistakenly 'rm -rf'ed the wrong directory, and hopefully it did not cause a huge damage.. Is there any way to prevent users from doing a similar unix horror story?? Someone mentioned (in the comments section of the previous link) that ... I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it ... Is there any implementation of that in any current Unix or Linux distro? And what is the common practice to prevent that error even from a sysadmin (with root access)? It seems that there was some protection for the root directory (/) in Solaris (since 2005) and GNU (since 2006). Is there anyway to implement the same protection way to some other folders as well?? To give it more clarity, I was not asking about general advice about rm usage (and I've updated the title to indicate that more), I want something more like the root folder protection: in order to rm -rf / you have to pass a specific parameter: rm -rf --no-preserve-root /.. Is there similar implementations for customized set of directories? Or can I specify files in addition to / to be protected by the preserve-root option? amyassin, Jan 20, 2013 at 17:26 I think pretty much people here mistakenly ' rm -rf 'ed the wrong directory, and hopefully it did not cause a huge damage.. Is there any way to prevent users from doing a similar unix horror story ?? Someone mentioned (in the comments section of the previous link ) that ... I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it ... Is there any implementation of that in any current Unix or Linux distro? And what is the common practice to prevent that error even from a sysadmin (with root access)? It seems that there was some protection for the root directory ( / ) in Solaris (since 2005) and GNU (since 2006). Is there anyway to implement the same protection way to some other folders as well?? To give it more clarity, I was not asking about general advice about rm usage (and I've updated the title to indicate that more), I want something more like the root folder protection: in order to rm -rf / you have to pass a specific parameter: rm -rf --no-preserve-root / .. Is there similar implementations for customized set of directories? Or can I specify files in addition to / to be protected by the preserve-root option? mattdm, Jan 20, 2013 at 17:33 1) Change management 2) Backups. – mattdm Jan 20 '13 at 17:33 Keith, Jan 20, 2013 at 17:40 probably the only way would be to replace the rm command with one that doesn't have that feature. – Keith Jan 20 '13 at 17:40 sr_, Jan 20, 2013 at 18:28 safe-rm maybe – sr_ Jan 20 '13 at 18:28 Bananguin, Jan 20, 2013 at 21:07 most distros do alias rm='rm -i' which makes rm ask you if you are sure. Besides that: know what you are doing. only become root if necessary. for any user with root privileges security of any kind must be implemented in and by the user. hire somebody if you can't do it yourself.over time any countermeasure becomes equivalaent to the alias line above if you cant wrap your own head around the problem. – Bananguin Jan 20 '13 at 21:07 midnightsteel, Jan 22, 2013 at 14:21 @amyassin using rm -rf can be a resume generating event. Check and triple check before executing it – midnightsteel Jan 22 '13 at 14:21 Gilles, Jan 22, 2013 at 0:18 To avoid a mistaken rm -rf, do not type rm -rf . If you need to delete a directory tree, I recommend the following workflow: • If necessary, change to the parent of the directory you want to delete. • mv directory-to-delete DELETE • Explore DELETE and check that it is indeed what you wanted to delete • rm -rf DELETE Never call rm -rf with an argument other than DELETE . Doing the deletion in several stages gives you an opportunity to verify that you aren't deleting the wrong thing, either because of a typo (as in rm -rf /foo /bar instead of rm -rf /foo/bar ) or because of a braino (oops, no, I meant to delete foo.old and keep foo.new ). If your problem is that you can't trust others not to type rm -rf, consider removing their admin privileges. There's a lot more that can go wrong than rm . Always make backups . Periodically verify that your backups are working and up-to-date. Keep everything that can't be easily downloaded from somewhere under version control. With a basic unix system, if you really want to make some directories undeletable by rm, replace (or better shadow) rm by a custom script that rejects certain arguments. Or by hg rm . Some unix variants offer more possibilities. • On OSX, you can set an access control list on a directory preventing deletion of the files and subdirectories inside it, without preventing the creation of new entries or modification of existing entries: chmod +a 'group:everyone deny delete_child' somedir (this doesn't prevent the deletion of files in subdirectories: if you want that, set the ACL on the subdirectory as well). • On Linux, you can set rules in SELinux, AppArmor or other security frameworks that forbid rm to modify certain directories. amyassin, Jan 22, 2013 at 9:41 Yeah backing up is the most amazing solution, but I was thinking of something like the --no-preserve-root option, for other important folder.. And that apparently does not exist even as a practice... – amyassin Jan 22 '13 at 9:41 Gilles, Jan 22, 2013 at 20:32 @amyassin I'm afraid there's nothing more (at least not on Linux). rm -rf already means "delete this, yes I'm sure I know what I'm doing". If you want more, replace rm by a script that refuses to delete certain directories. – Gilles Jan 22 '13 at 20:32 Gilles, Jan 22, 2013 at 22:17 @amyassin Actually, I take this back. There's nothing more on a traditional Linux, but you can set Apparmor/SELinux/ rules that prevent rm from accessing certain directories. Also, since your question isn't only about Linux, I should have mentioned OSX, which has something a bit like what you want. – Gilles Jan 22 '13 at 22:17 qbi, Jan 22, 2013 at 21:29 If you are using rm * and the zsh, you can set the option rmstarwait : setopt rmstarwait  Now the shell warns when you're using the * : > zsh -f > setopt rmstarwait > touch a b c > rm * zsh: sure you want to delete all the files in /home/unixuser [yn]? _  When you reject it ( n ), nothing happens. Otherwise all files will be deleted. Drake Clarris, Jan 22, 2013 at 14:11 EDIT as suggested by comment: You can change the attribute of to immutable the file or directory and then it cannot be deleted even by root until the attribute is removed. chattr +i /some/important/file This also means that the file cannot be written to or changed in anyway, even by root . Another attribute apparently available that I haven't used myself is the append attribute ( chattr +a /some/important/file . Then the file can only be opened in append mode, meaning no deletion as well, but you can add to it (say a log file). This means you won't be able to edit it in vim for example, but you can do echo 'this adds a line' >> /some/important/file . Using > instead of >> will fail. These attributes can be unset using a minus sign, i.e. chattr -i file Otherwise, if this is not suitable, one thing I practice is to always ls /some/dir first, and then instead of retyping the command, press up arrow CTL-A, then delete the ls and type in my rm -rf if I need it. Not perfect, but by looking at the results of ls, you know before hand if it is what you wanted. NlightNFotis, Jan 22, 2013 at 8:27 One possible choice is to stop using rm -rf and start using rm -ri . The extra i parameter there is to make sure that it asks if you are sure you want to delete the file. Probably your best bet with it would be to alias rm -ri into something memorable like kill_it_with_fire . This way whenever you feel like removing something, go ahead and kill it with fire. amyassin, Jan 22, 2013 at 14:24 I like the name, but isn't f is the exact opposite of i option?? I tried it and worked though... – amyassin Jan 22 '13 at 14:24 NlightNFotis, Jan 22, 2013 at 16:09 @amyassin Yes it is. For some strange kind of fashion, I thought I only had r in there. Just fixed it. – NlightNFotis Jan 22 '13 at 16:09 Silverrocker, Jan 22, 2013 at 14:46 To protect against an accidental rm -rf * in a directory, create a file called "-i" (you can do this with emacs or some other program) in that directory. The shell will try to interpret -i and will cause it to go into interactive mode. For example: You have a directory called rmtest with the file named -i inside. If you try to rm everything inside the directory, rm will first get -i passed to it and will go into interactive mode. If you put such a file inside the directories you would like to have some protection on, it might help. Note that this is ineffective against rm -rf rmtest . ValeriRangelov, Dec 21, 2014 at 3:03 If you understand C programming language, I think it is possible to rewrite the rm source code and make a little patch for kernel. I saw this on one server and it was impossible to delete some important directories and when you type 'rm -rf /direcotyr' it send email to sysadmin. #### bash - Safe Directory Delete ###### Mar 13, 2013 | Stack Exchange rm -i would get extremely annoying especially when removing recursively. rm -I, on the other hand seems a little more tolerable which will prompt once before removing more than three files, or when removing recursively. It's much less intrusive while still giving protection against most mistakes. The best form of protection is to double/triple check when deleting recursively or better yet, anything you delete. • In my experience, this kind of aliases just makes the "rm foo<ENTER>y<ENTER>" automatic... i.e., no protection at the cost of 2 extra keystrokes. And it isn't available everywhere. Better don't do that, except for the greenest training-wheel-needy users. – vonbrand Mar 13 '13 at 20:20 • Seeing that rm foo would only remove one file. rm foo would only ask once. If you're talking about when removing the directory foo/ then the command rm -Ir foo would ask once as well (if it contained more than 3 files). If you were to remove a directory recursively that had 1000 files in it (for arguments sake). Then you'd be there for quite a bit of time saying Yes to confirm all the files with -i where since the user should know what files their deleting -I will ask once for any removal operation with three or more files involved. – cinelli Mar 13 '13 at 22:41 • Adding to that. If the user doesn't know what's contained in foo/ then when rm -Ir foo is passed and it asks "You sure?" and the user says Yes without double checking and not knowing what's in the directory. Then, that's something nobody can answer other than. "Pay attention to what you're doing to you're system. Or go back to using the Recycle Bin in Windows." – cinelli Mar 13 '13 at 22:47 You can cd to the top level directory you want to remove recursively, then go up a level and rm that directory. That way you don't have to use leading slashes, etc. cd /home/user/junk ls -1 folders to delete rm -ri . ###Or "rm -rf ." cd .. rm -rf junk Another way to check that your paths are right for any command is to simply use tab completion. One option is to use a trash can approach, like here. There are a lot of examples out there besides this one - it was just the first in my search results. Basically, all rm operations become mv commands. If you rm the wrong files, they're still on the machine and easily recoverable. And the approach linked above creates an alias to the base rm command so you can still use it when you really want to. #### [Apr 22, 2018] bash - Is it safe to alias rm to safe-rm ###### Jun 7, 2014 | Ask Ubuntu Q: We are happy to use safe-rm as a replacement to rm, but is it considered safe to alias it? e.g. alias rm='safe-rm' A: Yes, an alias is just a shell-only rename. The scope of an alias is only the shell that you are currently in. safe-rm is pDELETE. Doing the deletion in several stages gives you an opportunity to verify that you aren't deleting the wrong thing, either because of a typo (as in rm -rf /foo /bar instead of rm -rf /foo/bar) or because of a braino (oops, no, I meant to delete foo.old and keep foo.new). If your problem is that you can't trust others not to type rm -rf, consider removing their admin privileges. There's a lot more that can go wrong than rm. Let me get into the problem you asked This tool, safe-rm, ships with a default system-wide configuration file (/etc/safe-rm.conf), but each user can supplement that list of "protected" directories and files by adding lines to ~/.safe-rm. Use \rm when you are sure about what you are deleting but most of the time, I prefer to go through the confirmation. Too many files disappeared because of a rushed removal action. #### THE classic Unix horror story WSU Linux Users Group *THE* classic Unix horror story Tue, 01/31/2006 - 10:11am - binford2k Try this in windows! #### Unix Recovery Legend #### This classic article from Mario Wolczko first appeared on Usenet in 1986. Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters.... It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg. My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed grep 147 /etc/passwd only to receive the response /etc/passwd: No such file or directory. Instantly, I guessed that something was amiss. This was confirmed when in response to ls /etc I got ls: not found. I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager. When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines: # cd # rm -rf * Oh, shit, I thought. That would just about explain it. I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and typing cd / echo * I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev,  /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched. Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost  /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know[work?] without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck. Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable. So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd,  /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack. By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working. Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference: 070100002c000000000000000000000000000000000000000000000000000000 0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f 8800040000bc012f65746300 I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved. So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history. Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one. The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat. And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together. I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to: Mario Wolczko ------------------------------------------------------------------------ Dept. of Computer Science ARPA: miw%uk.ac.man.cs.ux@cs.ucl.ac.uk The University USENET: mcvax!ukc!man.cs.ux!miw Manchester M13 9PL JANET: miw@uk.ac.man.cs.ux U.K. 061-273 7121 x 5699 ------------------------------------------------------------------------ Re: *THE* classic Unix horror story We used to outsource some of our support at our data center, and we had multiple rm's from those agents. Luckily, live CD's can get you back rolling to where you can rsync from backups. # cd # rm -rf * If I'm not mistaken, wouldn't this just delete everything in /root ? I'm more of a Linux guy, but I'd presume you'd need the following command to wipe everything. rm -rf / -Chris | Editor Mozy Reviews Mon, 04/25/2011 - 3:43pm - binford2k Re: *THE* classic Unix horror story It all depends on where root's home dir is. Some unixes do have it set to / Thu, 05/19/2011 - 9:36am - mchristenson Re: *THE* classic Unix horror story It doesn't matter whether you keep anything in /root or /home, if the distribution you are using has the root user's HOME environment variable set to /. POSIX standard says that, when not supplied with any arguments, the cd command will change to the directory pointed to by HOME (or an implementation based location if HOME is unset or undefined.) If the former, subsequently invoking rm -rf * would then remove both your /root and /home directories. Mon, 01/03/2011 - 12:39am - scott.allerdice Re: *THE* classic Unix horror story Not classic horrible yes, the admins did a good job of killing the process before it made it's way to the rest of the file system. I had a similar problem on windows in the 90's when malware didn't exist and viruses weren't bots but destroyed files. I went to start run telnet and it started searching for telnet.exe i knew something was wrong so went to my windows folder to look for it manually and didn't see it. How strange i thought i just saw a file disappear and another etc. Yup files were disappearing before my very eyes. It was as bad as rm -rf and i had no time to look and see what was doing it so i pulled the plug on my machine. It's what i would do had i seen rm -rf. Pull the power and figure out a plan. For windows i had norton on floppy and was able to clean it out i don't know what i lost but got the system back up and running. Unix wise i've ran redhat, slack, suse, ubuntu, freebsd, and sparc x86 i make it a point to have weekly backups of my drives and with out using recovery CD's or programs i do it almost manually copying files over to the drives so i know they are in a format i can access once recovered. If a friend ever threatened an rm -rf for$5 i would comply however his next trip home would be memorable i would fill his suite case with sex toys and synthetic marijuana and coke powder, it's legal and used to train drug sniffing dogs. It would make for a hell of a trip through TSA screening and possibly a trip to jail court or maybe even drug rehab i'd tell him he can add the $5 to the bail money i'd put up for him. Re: *THE* classic Unix horror story Heh, ingenious and creative. I would have given up about half way through (if that) Re: *THE* classic Unix horror story This is without a doubt the most convoluted restore process I've ever heard of. It sounds to me like the operators involved were too naïve when it came to the install process for 4.3BSD. It really isn't that involved. Honestly, you have to wonder who installed their OS, and why didn't they have access to the manuals? I mean even if they paid for the tape only, wouldn't they have printed out the system manuals? I could have had that box up & running again in under an hour. First thing to do would be to locate the install tape, and halt the machine. Boot from tape, and run the copy program to install the miniroot into the swap partition. Rebooting the VAX again, into the swap partition I would restore the root partition from one week old root backup. One more reboot, and the VAX would have it's root back from their best backup, then it would be a matter of restoring other directories as needed. Oh and yes by default 4.3BSD has separate root partition from the rest of the userland stuff. I can only think that these people were absolutely terrified of rebooting their VAX. I wonder if it was an 11/780 with a borked console tape/floppy. But who knows this is over 20 years ago. Why not use tar One way to avoid the assembler is to use a combination of tar and uuencode Basically, if you transfer a tar file with only one directory in it (by manual typing) - you would be able to extract it using tar xvf etc.tar -C / Only issue is that the size is a bit larger - but I think if one gives enough thought to this - it could be reduced. The key to this ofcourse - as you say, is to NOT PANIC. Fri, 03/14/2008 - 3:11pm - joeblow1470 Re: Why not use tar tar is located at /bin/tar as it states in the story(about 1/3 of the way down)(bin was one of the deleted directories) PS great story Thu, 10/07/2010 - 10:51am - jfwfmt Re: *THE* classic Unix Horror story not on Unix, on RT-11 but a close equivalent horror story: One night I was finishing up the code for the VAX 8600 console. I had finally made the changes to the OS code and to the clock driver to translate the 64Hz interrupt rate to the 60Hz clock on the PDP-11 (with a little studder every 1/4 second) and to transfer the 1KHz clock through the mismatched registers windows to the VAX. So everything was good and i decided to clean up and leave. I entered: . delete *,tmp It took a long time and then came back with: tmp file not found. Of course the comma rather than the intended dot made all the difference. . directory returned with nothing. Not to worry, there were men in white coats who religiously took backups of everything every night, Except they had never done a restore. Which of course failed. So I came in the next day and started over. Fri, 12/10/2010 - 10:00pm - greffedecheveux (not verified) #### [Apr 21, 2018] Any alias of rm is a very stupid idea ##### Option -I is more modern and more useful then old option -i. It is highly recommended. And it make sense to to use alias with it contrary to what this author states (he probably does not understand that aliases do not wok for non-interactive sessions.). ##### The point the author make is that when you automatically expect rm to be aisles to rm -i you get into trouble on machines where this is not the case. And that's completely true. ##### But it does not solve the problem as respondents soon became automatic. stated. Writing your own wrapper is a better deal. One such wrapper -- safe-rm already exists and while not perfect is useful ##### Notable quotes: ##### "... A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do. ..." ##### "... It you want a safety net, do "alias del='rm -I –preserve_root'", ..." ###### Feb 14, 2017 | www.cyberciti.biz Art Protin June 12, 2012, 9:53 pm Any alias of rm is a very stupid idea (except maybe alias rm=echo fool). A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do. It you want a safety net, do "alias del='rm -I –preserve_root'", Drew Hammond March 26, 2014, 7:41 pm ^ This x10000. I've made the same mistake before and its horrible. #### [Jun 12, 2010] Sysadmin Blunder (3rd Response) ###### Toolbox for IT Groups jxtmills: I needed to clear a directory full of ".directories" and I issued: rm -r .* It seemed to find ".." awfully fast. That was a bad day, but, we restored most of the files in about an hour. John T. Mills alain: hi every one for me , i remember 2 1 - 'rm *;old' in / directory note ';' instead of '.' 2 - kill #pid of the informix process (oninit) and delete it (i dreamed) jturner: Variation on a theme ... the 'rm -r theme' as a junior admin on AIX3.2.5, I had been left to my own devices to create some housekeeping scripts - all my draft scripts being created and tested in my home directory with a '.jt' suffix. After completing the scripts I found that I had inadvertantly placed some copies in / with a .jt suffix. Easy job then to issue a 'rm *.jt' in / and all would be well. Well it would have been if I hadn't put a space between the * and the .jt. And the worst thing of all, not being a touch typist and looking at the keys, I glanced at the screen before hitting the enter key and realised with horror what was going to happen and STILL my little finger continued to proceed toward the enter key. Talk about 'Gone in 60 seconds' - my life was at an end - over - finished - perhaps I could park cars or pump gas for a living. Like other correspondents a helpful senior admin was on hand to smile kindly and show me how to restore from mksysb-fortunately taken daily on these production systems. (Thanks Pete :))) ) To this day, rm -i is my first choice with multiple rm's just as a test!!!!!! Happy rm-ing :) daguenet: I know that one. Does any body remember when the rm man page had a warning not do rm -rf / as root? How may systems were rebuild due that blunder. Not that I have ever done something like that, nor will ever admit to it:). Aaron cal.staples: That is a no brainer! First a little background. I cooked up a script called "killme" which would ask for a search string then parse the process table and return a list of all matches. If the list contained the processes you wanted to kill then you would answer "Yes", not once, but twice just to be sure you thought about it. This was very handy at times so I put it out on all of our servers. Some time had passed and I had not used it for a while when I had a need to kill a group of processes. So I typed the command not realizing that I had forgotten the scripts name. Of course I was on our biggest production system at that time and everything stopped in it's tracks! Unknown to me was that there is an AIX command called "killall" which is what I typed. From the MAN page: "The killall command cancels all processes that you started, except those producing the killall process. This command provides a convenient means of canceling all processes created by the shell that you control. When started by a root user, the killall command cancels all cancellable processes except those processes that started it." And it doesn't ask for confirmation or anything! Fortunately the database didn't get corrupted and we were able to bring everything back on line fairly quickly. Needless to say we changed the name of this command so it couldn't be run so easily. "Killall" is a great command for a programmer developing an application which goes wild and he/she needs to kill the processes and retain control of the system, but it is very dangerous in the real world! Jeff Scott: The silliest mistake? That had to be a permissions change on /bin. I got a call from an Oracle DBA that the$ORACLE_HOME/bin no longer belonged to
oracle:dba. We never found out how that happened. I logged in to change the
permissions. I accidentally typed cd /oracle.... /bin (note the space
before /bin), then cheerfully entered the following command:

#chown -R oracle:dba ./*

The command did not climb up to root fortunately, but it really made a mess.
We ended up restoring /bin from a backup taken the previous evening.

Jeff Scott
Darwin Partners/EMC

Cal S.

tzhou:

crontab -r when I wanted to do crontab -e. See letters e and r are side by
side on the keyboard. I had 2 pages of crontab and had no backup on the
machine !

Jeff Scott:

I've seen the rm -fr effect before. There were no entries in any crontab.
Before switching to sudo, the company used a homegrown utility to grant
things root access. The server accepted ETLs from other databases, acting as
a data warehouse. This utility logged locally, instead of logging via
syslogd with local and remote logging. So, when the system was erased, there
really was no way to determine the actual cause. Most of the interactive
users shared a set of group accounts, instead of enforcing individual
accounts and using su - or sudo. The outage cost the company $8 million USD due to lost labor that had to be repeated. Clearly, it was caused by a cleanup script, but it is anyone's guess which one. Technically, this was not a sysadmin blunder, but it underscores the need for individual accounts and for remote logging of ALL uses of group accounts, including those performed by scripts. It also underscores the absolute requirement that all scripts have error trapping mechanisms. In this case, the rm -fr was likely preceded by a cd to a nonexistent directory. Since this was performed as root, the script cd'd to / instead. The rm -fr then removed everything. The other possibility is that it applied itself to another directory, but, again, root privileges allowed it to climb up the directory tree to root. #### Creative uses of rm ##### See also Syaadmin Horror Stories • From: broadley@neurocog.lrdc.pitt.edu (Bill Broadley) Organization: University of Pittsburgh I was deleting last semesters users to try to dig up some disk space, I also deleted some test users at the same time. One user took longer then usual, so I hit control-c and tried ls. "ls: command not found" Turns out that the test user had / as the home directory and the remove user script just happily blew away the whole disk. ftp, telnet, rcp, rsh, etc were all gone. Had to go to tapes, and had one LONG rebuild of X11R5. Fortunately it wasn't our primary system... • From: cjc@ulysses.att.com (Chris Calabrese) Organization: AT&T Bell Labs, Murray Hill, NJ, USA We have a home-grown admin system that controls accounts on all of our machines. It has a remove user operation that removes the user from all machines at the same time in the middle of the night. Well, one night, the thing goes off and tries to remove a user with the home directory '/'. All the machines went down, with varying amount of stuff missing (depending on how soon the script, rm, find, and other importing things were clobbered). Nobody knew what what was going on! The systems were restored from backup, and things seemed to be going OK, until the next night when the remove-user script was fired off by cron again. This time, Corporate Security was called in, and the admin group's supervisor was called back from his vacation (I think there's something in there about a helicopter picking the guy up from a rafting trip in the Grand Canyon). By chance, somebody checked the cron scripts, and all was well for the next night... • From: tzs@stein.u.washington.edu (Tim Smith) Organization: University of Washington, Seattle I was working on a line printer spooler, which lived in /etc. I wanted to remove it, and so issued the command "rm /etc/lpspl." There was only one problem. Out of habit, I typed "passwd" after "/etc/" and removed the password file. Oops. I called up the person who handled backups, and he restored the password file. A couple of days later, I did it again! This time, after he restored it, he made a link, /etc/safe_from_tim. About a week later, I overwrote /etc/passwd, rather than removing it. After he restored it again, he installed a daemon that kept a copy of /etc/passwd, on another file system, and automatically restored it if it appeared to have been damaged. Fortunately, I finished my work on /etc/lpspl around this time, so we didn't have to see if I could find a way to wipe out a couple of filesystems... • From: bill@chaos.cs.umn.edu ( bill pociengel) Organization: University of Minnesota After a real bad crash (tm) we got to test our backup by doing: # cd / # rm -rf * ohhhhhhhh sh*t i hope those tapes are good. Ya know it's kinda funny (in a perverse way) to watch the system just slowly go away. • From: barrie@calvin.demon.co.uk (Barrie Spence) Organization: DataCAD Ltd, Hamilton, Scotland My mistake on Sun was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed "rm -rf ./.*". I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first. • From: robjohn@ocdis01.UUCP (Contractor Bob Johnson) Organization: Tinker Air Force Base, Oklahoma Cleaning out an old directory, I did 'rm *', then noticed several files that began with dot (.profile, etc) still there. So, in a fit of obtuse brilliance, I typed... rm -rf .* & By the time I got it stopped, it had chewed through 3 filesystems which all had to be restored from tape (.* expands to ../*, and the -r makes it keep walking up the directory tree). Live and learn... • From: JRowe@cen.ex.ac.uk (John Rowe) Organization: Computer Unit. - University of Exeter. UK rik@nella15.cc.monash.edu.au (Rik Harris) writes: [snippet about "using 'find' in an auto-cleanup script which blew away half of the source" deleted. -ed.] If you're doing this using find always put -xdev in: find /tmp/ -xdev -fstype 4.2 -type f -atime +5 -exec rm {} \; This stops find from working its way down filesystems mounted under /tmp/. If you're using, say, perl you have to stat . and .. and see if they are mounted on the same device. The fstype 4.2 is pure paranoia. Needless to say, I once forgot to do this. All was well for some weeks until Convex's version of NQS decided to temporarily mount /mnt under /tmp... Interestingly, only two people noticed. Yes, the chief op. keeps good backups! Other triumphs: I created a list of a user's files that hadn't been accessed for three months and a perl script for him to delete them. Of course, it had to be tested, I mislaid a quote from a print statement... This did turn into a triumph, he only wanted a small fraction of them back so we saved 20 MB. I once deleted the only line from within an if..then statement in rc.local, the sun refused to come up, and it was surprisingly difficult to come up single user with a writeable file system. AIX is a whole system of nightmares strung together. If you stray outside of the sort of setup IBM implicitly assume you have (all IBM kit, no non IBM hosts on the network, etc.) you're liable to end up in deep doodoo. One thing I would like all vendors to do (I know one or two do) is to give root the option of logging in using another shell. Am I the only one to have mangled a root shell? • From: rheiger@renext.open.ch (Richard H. E. Eiger) Organization: Olivetti (Schweiz) AG, Branch Office Berne Just imagine having the sendmail.cf file in /etc. Now, I was working on the sendmail stuff and had come up with lots of sendmail.cf.xxx which I wanted to get rid of so I typed "rm -f sendmail.cf. *". At first I was surprised about how much time it took to remove some 10 files or so. Hitting the interrupt key, when I finally saw what had happened was way to late, though. Fortune has it that I'm a very lazy person. That's why I never bothered to just back up directories with data that changes often. Therefore I managed to restore /etc successfully before rebooting... :-) Happy end, after all. Of course I had lost the only well working version of my sendmail.cf... • From: gfowler@javelin.sim.es.com (Gary Fowler) Organization: Evans & Sutherland Computer Corporation Once I was going to make a new file system using mkfs. The device I wanted to make it on was /dev/c0d1s8. The device name that I used, however, was /dev/c0d0s8 which held a very important application. I had always been a little annoyed by the 10 second wait that mkfs has before it actually makes the file system. I'm sure glad it waited that time though. I probably waited 9.9 seconds before I realized my mistake and hit that DEL key just in time. That was a near disaster avoided. [ I wish all systems were like that. Linux mkfs doesn't wait, but at ] [ least I have the source! -ed. ] Another time I wasn't so lucky. I was a very new SA, and I was trying to clean some junk out of a system. I was in /usr/bin when I noticed a sub directory that didn't belong there. A former SA had put it there. I did an ls on it and determined that it could be zapped. Forgetting that I was still in /usr/bin, I did an rm *. No 10 second idiot proofing with rm. Now if some one would only create an OS with a "Do what I mean, not what I say" feature. Gary "Experience is what allows you to recognize a mistake the second time you make it." Fowler • From: russells@ccu1.aukuni.ac.nz (Russell Street) Organization: University of Auckland, New Zealand. I once had "gnu-emacs" aliased to 'em' (and 'emacs' etc) One day I wanted to edit the start up file and mistyped # rm /etc/rc.local instead of the obvious. *Fortunately* I had just finished a backup and was now finding out the joys of tar and it's love of path names. [./etc/rc.local and /etc/rc.local and etc/rc.local) are *not* the same for tar and TK-50s take a *long* time search for non-existant files :(] Of course the BREAK (Ctrl-P) key on a VAX and an Ultrix manual and a certain /etc/ttys line are just a horror story waiting to happen! Especially when the VAX and manuals are in a unsupervised place :) • From: rik@nella15.cc.monash.edu.au (Rik Harris) Organization: Monash University, Melb., Australia. Most of our disks reside on a single, high-powered server. We decided this probably wasn't too good an idea, and put a new disk on one of the workstations (particularly since the w/s has a faster transfer rate than the server does!). It's still really useful to be able to use all disks from the one machine, so I mounted the w/s disk on the server. I said to myself (being a Friday afternoon...see previous post) "it's only temporary.../mnt is already being used...I'll mount it in /tmp". So, I mounted on /tmp/a (or something). This was fine for a few hours, but then the auto-cleanup script kicked in, and blew away half of my source (the stuff over 2 weeks old). I didn't notice this for a few days, though. After I figured out what had happened, and restored the files (we _do_ have a good backup strategy), everything was OK. Until a few months later. We were trying to convince a sysadmin from another site that he shouldn't NFS export his disks rw,root to everyone, so I mounted the disk to put a few suid root programs in his home directory to convince him. Well, it's only a temporary mount, so.... You guessed it, another Friday afternoon. I did a umount /tmp/b, and forgot about it. I noticed this one about halfway through the next day. (NFS over a couple of 64k links is pretty slow). The disk had not unmounted because it was busy...busy with two find scripts, happily checking for suid programs, and deleting anything over a week old. A df on the filesystem later showed about 12% full :-( Sorry Craig. Now, I create /mnt1, /mnt2, /mnt3.... :-) Remember....Friday afternoons are BAD news. • From: ranck@joesbar.cc.vt.edu (Wm. L. Ranck) Well, after reading some of the stories in this thread I guess I can tell mine. I got an RS/6000 mod. 220 for my office about 6 months ago. The OS was preloaded so I had little chance to learn that process. Being used to a full-screen editor I was not happy with vi so I read in the manual that INED (IBM's editor for AIX) was full-screen and I logged in as root and installed it. I immediately started to play with the new editor and somehow found a series of keys that told the editor to delete the current directory. To this day I don't know what that sequence of keys was, but I was unfortunately in the /etc directory when I found it, and I got a prompt that said "do you want to remove this?" and I thought i was just removing the file I had been playing with but instead I removed /etc! I got the chance to learn how to install AIX from scratch. I did reinstall INED even though I was a little gun-shy but I made sure that whenever I used it from then on I was *not* root. I have since decided that EMACS may be a better choice. • From: root@rulcvx.LeidenUniv.nl (root) Organization: CRI, institute for telecommunication and computerservices. Well, waddya know... Some half hour ago, coming back from root (I was installing m4 on our system) [Shit, all my neato emacs tricks won't work. Damn, damn, damn kill, kill, KILL] to my own userid, I got this little message: "Can't find home directory /mnt0/crissl." and an other: "Can't lstat .". [Grrrrr, ^S and ^Q haven't been remapped...] Guess what happened, not an hour ago... A collegue of mine was emptying some directories of computer-course accounts. As I did a "ps -t" on his tty, what did I see? "rm -rf .*" Well, I'm not alone, he got sixteen other homedirectories as well. And guess what filesystems we don't make incremental backups of... And why not? Beats me... I haven't killed him yet, he first has to restore the lot. And for those "touch \-i" fans out there: you wouldn't have been protected... • From: jcm@coombs.anu.edu.au (J. McPherson) Organization: Australian National University A few months ago in comp.sys.hp, someone posted about their repairs to an HP 7x0, after a new sysadmin had just started work. They {the new person} had been looking throught the file system to try to make some space, saw /dev and the mainly 0 length files therein. Next command was "rm -f /dev/*" and they wondered why they couldn't login ;) I think the result was that the new person was sent on a sysamin's course a.s.a.p • From: msb@sq.sq.com (Mark Brader) Organization: SoftQuad Inc., Toronto, Canada > ... if you're trying rm -rf / you'll NEVER get a clear disk - at least > /bin/rm (and if it reached /bin/rmdir before scanning some directories > then add a lot of empty directories). I've seen it once... Then it must be version-dependent. On this Sun, "cp /bin/rm foo" followed by "./foo foo" does not leave a foo behind, and strings shows that rm appears not to call rmdir (which makes sense, as it can just use unlink()). In any case, I'm reminded of the following article. This is a classic which, like the story of Mel, has been on the net several times; it was in this newsgroup in January. It was first posted in 1986. Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters.... It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg. My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed grep 147 /etc/passwd only to receive the response /etc/passwd: No such file or directory. Instantly, I guessed that something was amiss. This was confirmed when in response to ls /etc I got ls: not found. I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager. When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines: # cd # rm -rf * Oh, shit, I thought. That would just about explain it. I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and typing cd / echo * I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched. Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck. Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable. So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack. By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working. Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference: 070100002c000000000000000000000000000000000000000000000000000000 0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f 8800040000bc012f65746300 I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved. So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history. Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one. The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat. And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together. I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to: Mario Wolczko • From: samuel@cs.ubc.ca (Stephen Samuel) Organization: University of British Columbia, Canada Some time ago, I was editing our cron file to remove core more than a day old. Unfortunately, thru recursing into VI sessions, I ended up saving an intermediate (wron) version of this file with an extra '-o' in it. find / -name core -o -atime +1 -exec /bin/rm {} \; The cute thing about this is that it leaves ALL core files intact, and removes any OTHER file that hasn't been accessed in the last 24 hours. Although the script ran at 4AM, I was the first person to notice this, in the early afternoon.. I started to get curious when I noticed that SOME man pages were missing, while others were. Up till then, I was pleased to see that we finally had some free disk space. Then I started to notice the pattern. Really unpleasant was the fact that no system backups had taken place all summer (and this was a research lab). The only saving grace is that most of the really active files had been accessed in the previous day (thank god I didn't do this on a saturday). I was also lucky that I'd used tar the previous day, as well. I still felt sick having to tell people in the lab what happened. • From: Stephen Samuel (samuel@cs.ubc.ca) Organization: University of British Columbia, Canada As some older sys admins may remember, BSD 4.1 used to display unprintable characters as a questionmark. An unfortunate friend of mine had managed to create an executable with a name consisting of a single DEL character, so it showed up as "?*". He tried to remove it. "rm ?*" he was quite frustrated by the time he asked me for help, because he had such a hard time getting his files restored. Every time he walked up to a sys-admin type and explained what happened, they'd go "you did WHAT?", he'd explain again, and they'd go into a state of uncontrolable giggles, and he'd walk away. I only giggled controlably. This was at a time (~star wars) when it was known to many as "the mythical rm star". • From: jjr@ctms.gwinnett.com (J.J. Reynolds) Organization: Consolidated Traffic Management Services (CTMS) The SCO man page for the rm command states: It is also forbidden to remove the root directory of a given file system. Well, just to test it out, I one day decided to try "rm -r /" on one of our test machines. The man page is correct, but if you read carefully, it doesn't say anything about all of the files underneath that filesystem.... • From: bcutter@pdnis.paradyne.com (Brooks Cutter) A while back I installed System V R4 on my 386 at home for development purposes... I was compiling programs both in my home directory, and in /usr/local/src ... so in order to reduce unnecessary disk space I decided to use cron to delete .o files that weren't accessed for over a day... I put the following command in the root cron... find / -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \; (instead of putting) find /home/bcutter -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \; find /usr/local/src -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \; The result was that a short time later I was unable to compile software. What the first line was doing was zapping the files like /usr/lib/crt1.o .. and later I found out all the Kernel object files... OOPS! After this happened a second time (after re-installing the files from tape) I tracked down the offending line and fixed it.... Yet another case of creating work by trying to avoid extra work (in this case a second find line). #### There's an interesting thread on the CentOS mailing list on sys admin horror stories. Amusing read. http://lists.centos.org/pipermail/centos/2010-June/095423.html #### [Ilugc] [OT] sys admin horror stories I once had a vendor sys admin run an "rm -rf ./project/ subdir" and delete the entire ./project folder and 2 months of live data. Same vendor had also not noticed that automated backups had failed for many weeks and we were running without any backup for a long time. A penalty of Rs 2 lakhs was enforced, all because of one extra space in the command. - Raja #### Greatest blunders ###### IT Resource Center Christian Gebhardt Hi As a newby in UNIX I had an Oracle Testinstallion on a production system productiv directory: /u01/... test directory: /test/u01/... deleting the test installation: cd /test rm /u01 OOPS ... After several bdf commands I noticed that the wrong lvol shrinks and stops the delete command with Ctrl'C The database still worked without the most binaries and libraries and after a restore from tape without stopping and starting the database all was ok. I love oracle ;-) Chris harry d brown jr Learning hpux? Naw, that's not it....maybe it was learning to spell aix?? sco?? osf?? Nope, none of those. The biggest blunder: One morning I came in at my usual time of 6am, and had an operator ask me what was wrong with one of our production servers (servicing 6 banks). Well nothing worked at the console (it was already logged in as root). Even a "cat *" produced nothing but another shell prompt. I stopped and restarted the machine and when it attempted to come back up it didn't have any OS to run. Major issue, but we got our backup tapes from that night and restored the machine back to normal. I was clueless (sort of like today) The next morning, the same operator caught me again, and this time I was getting angry (imagine that). Same crap, different day. Nothing was on any disk. This of course was before we had raid availble (not that that would have helped). So we restored the system from that nights backups and by 8am the banks have their systems up. So now I have to fix this issue, but where the hell to start? I knew that production batch processing was done by 9PM, and that the backups started right after that. The backups completed around 1am, which were good backups, because we never lost a single transaction. But around 6am the stuff hit the fan. So I had a time frame: 1am-6am, something was clobbering the system. I went though the crons, but nothing really stood out, so I had to really dive into them. This is the code (well almost) I found in the script: cd /tmp/uniplex/trash/garbage rm -rf * As soon as I saw those two lines, I realized that I was the one that had caused the system to crap out every morning. See, I needed some disk space, and I was doing some house cleaning, and I deleted the sub-directory "garbage" from the /tmp/uniplex/trash" directory. Of course the script is run by root, which attempted to "CD" to a non-existent directory, which failed, and cron was still cd'd to "/", it then proceeded to "rm -rf *" my system! live free or die harry Bill Hassell I guess my blunder sets the record for "most clobbered machines" in one day: I created an inventory script to be used in the Response Center to track all the systems across the United States (about 320 systems). These are all test and problem replication machines but necessary for the R/C engineers to replicate customer problems. The script was written about 1992 to handle version 7.0 and higher. About 1995, I had a number of useful scripts that it seemed reasonable to drop these into all 300 machines as a part of the inventory process (so far, so good). Then about that time, 10.01 was released and I made a few changes to the script. One was to change the useful script location from /usr/local/bin to /usr/contrib/bin because of bad directory permissions. I considered 'fixing' the bad permissions but since these systems must represent the customer environment, I decided to move everything. Enter the shell option -u. I did not use that option in my scripts and due to a spelling error, an environment variable was used in rm -r which was null, thus removing the entire /usr/local directory on 320 machines overnight. Needless to say, I never write scripts without set -u at the top of the script. Dave Johnson Here is my worst. We us BC's on our XP512. We stop the application, resync the BC, split the BC, start the application, mount the BC on same server, start backup to tape from BC. Well I had to add a LUN to the primary and BC. I recreated the BC. I forgot to change the script that mounts the BC to include the new LUN. The error message vgimport when you do not include all the LUN's is just a warning and it makes the volume group available. The backups seemed to be working just fine. Well 2 months go by. I did not have enough available disk space to test my backups. (That has been changed). Then I decided to be proactive about deleted old files. So I wrote a script: cd /the/directory/I/want/to/thin/out find . -mtime +30 -exec rm {} \; Well that was scheduled on cron to run just before backups one night. The next morning I get the call the system is not responding. (I guessed later the cd command had failed and the find ran from /). After a reboot I find lots of files are missing from /etc /var /usr /stand and so on. No problem, just rebuild from the make_recovery tape created 2 nights before then restore the rest from backup. Well step 1 was fine, but the backup tape was bad. The database was incomplete. It took 3 days (that is 24 hours per day) to find the most recient tape with a valid database. Then we had to reload all the data. After the 3rd day I was able to turn over recovery to the developers. It took about a week to get the application back on-line. I have sent a request to HP to have the vgimport command changed so a vgimport that does not specify all the LUN's will fail unless some new command line param is used. They have not yet provided this "enhancement" as of the last time I checked a couple of months ago. I now test for this condition and send mail to root as well as fail the BC mount if it does. Life lesson: TEST YOUR BACKUPS!! RAC Well I was very very new to HP-UX. Wanted to set up PPP connection with a password borrowed from a friend so that I could browse the net. Did not worry that the remote support modem can not dial out from remote support port. Went through all documents available, created device files dozen times, but never worked. In anguish did rm -fr ltr|tail -4|Awk '{print$9}'

(That to pacify myself that I know complex commands)

But alas, I was /sbin/rc3.d.

Thought this is not going to work and left that.
Other colleage not aware of this rebooted the system for Veritas netbackup problem.

Within next two hours HP engineer was on-site. Was called by colleague.

Was watching whole recovery process, repeatedly saying "I want to learn, I want to learn"

Then came to know that can not be done.

RD

Ramkumar Devanathan

I deleted /etc/* once. (just a few config files, eh?? nope, had to reinstall the OS);-)

Dario

Got a page once at 4:00AM about a filesystem almost full. Got up and started working on it without checking if someone else was in the system. I was recalling my commands using ESC-k but my supervisor was removing files in a different directory using rm -r *. The rest is history.

Gary Cantwell

A trainee student of one of my customers deleted /sbin on a HP-UX 11 system and copied it back in from a HP-UX 10.20 system and tried to deny it...

He's no longer a trainee student :-)

Gary

YLTan

got another one;

Trying to be helpful....I use a find . -name *.log -exec rm -f {} \; to rid all unused log files in apps temp and log dir. but I got a bit creative and put this in root cronjobs and guess what!! the next day, some very angry DBA come banging...cos it removed all Oracle database re-do log files.

Spend the rest of the weekend restoring database....:(

### Sites

bash - Safe Directory Delete - Unix & Linux Stack Exchange

Cleaning Up Your -tmp...The Safe Way Issue 18

Is it safe to delete stuff in /tmp (SUSE 10.2)

delete - Can I safely remove all the files in /tmp - Ask Ubuntu

linux - Delete all of -var-log - Server Fault

## Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least

Copyright © 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

 You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.