Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells

Creative uses of rm with unintended consequences

Wiping out much more then you intended to do

News Enterprise Unix System Administration Recommended Links Simple Unix Backup Tools Unix rm command Unix mv command
Missing backup horror stories Mistakes made because of the differences between various Unix/Linux flavors Creative uses of rm Lack of testing complex, potentially destructive, commands before execution of production box Pure stupidity Performing the operation on a wrong server
Safe-rm Typical Errors In Using Find Tips Unix History Humor Etc

Introduction

Many Unix sysadmin horror stories are related to unintended consequences such as side effects of classic Unix commands. One of the most prominent of such commands is rm.   Which has several idiosyncrasies which are iether easily forgotten or unknown by novice sysadmins. For example behavior of rm -r .* can easily be forgotten from one encounter to another, especially if sysadmin works both with Unix and Windows servers. Consequences if executed as root are tragic...  

While most, probably 99.9% of operations with rm are successful it is the remaining.01% that create a lot of damage and is subject to sysadmin folklore.

The command rm has relatively few options.  Two of the then are especially important:

Linux uses GNU implementation that as of April 2018 has the following options (note relatively new option -I):

-f, --force
ignore nonexistent files, never prompt
-i
prompt before every removal
-I
prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes
--interactive[=WHEN]
prompt according to WHEN: never, once (-I), or always (-i). Without WHEN, prompt always
--one-file-system
when removing a hierarchy recursively, skip any directory that is on a file system different from that of the corresponding command line argument
--no-preserve-root
do not treat '/' specially
--preserve-root
do not remove '/' (default)
-r, -R, --recursive
remove directories and their contents recursively
-v, --verbose
explain what is being done
--help
display this help and exit
--version
output version information and exit

Wiping out useful data by using rm with wrong parameters or options is probably  the most common blunder by both novice and seasoned sysadmins. It happens rarely but the result are often devastating.  This danger can't be completely avoided, but in addition to having an up-to-date backup, there are several steps that you can do to reduce the chances of the major damage to the data:

  1. Do backup before any dangerous operation. That the Rule no. 1
  2. You can make /usr directory write-protected by using a separate filesystem for it (/usr/local should be symlinked to /opt where it belongs). There can be some other write protected filesystem as appropriate. 
  3. Try to use option -I (instead of -i). While -i option (which it default alas in Red Hat and other distributions) is annoying and most often user disable it, there is more modern and more useful option -I which is highly recommended. Unfortunately very few sysadmin know about its existence. It prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes (rm(1) remove files-directories ):

    If the -I or --interactive=once option is given, and there are more than three files or the -r, -R, or --recursive are given, then rm prompts the user for whether to proceed with the entire operation. If the response is not affirmative, the entire command is aborted.

  4. Block operation for system level 2 directories such  as /etc or /usr (this simple measure actually is now implemented in GNU rm used in Linux) using wrapper such  as safe-rm. Aliasing rm to wrapper is safe because aliases work only in interactive sessions. In addition you can block the deletion of directories that are in your favorite directories list if you have any.  For example if you use level 2 directory /Fav for links some often used directories, you can check is the directory you are trying to delete is present in those links.
  5. Create list of files and  directories that  should not be erased and use it in wrapper. Instead of creating  the list you can mark them in some way to distinguish  from other. Linux does not have system attribute for files, but sticky bit on files can be used instead or along with ownership of sys instead of root).  You can also use extended attributes. You can also use file stamps using special meaning for seconds  field in the data stamp like in DOS: any file with creation date that have 00 seconds in it should never be deleted. If you operate server with SE linux in enforcing mode you can use SElinux for file protection.
  6. Rename the directory before applying rm -r command to a special name DELETEIT or some other mnemonic  name and then deleting it using special alias dirdel or something like that (this way you will never mistype the name of the directory): many devastating blunders with rm  occur when you think that you operate on backup copy of the  system or important user/application directory, but in reality operate on a "real thing" or by  accidentally typing "/" inform on name of common system directories such as /etc and  /var
  7. Write the command in editor first. Do not improvise with such command on the command line. A weaker version of this rule is to write options  first and after verification add rm in front.   Sysadmins  automatically type the name of the most common systems directory with slash in front of it  instead of proper relative name (rm -r /etc instead of rm -r etc ) because they are conditioned to type it this way by a long practice and this sequence of keystroke is "etched" in your brain and you do it almost semi-consciously. If you first rename the directory to something else, for example DELETEIT first,  such an error will not occur . A script can check if there is another directory with this name in existence and warn you too.  You can also simply move directory to /Trash folder which has, say 90 day expiration period for files in it
  8. Use a protective script as an envelope to rm. It can  that take into account some of the most dangerous situation typical in your environment. You can write a script like rmm or saferm  which among other useful preventive checks introduces a delay between hitting enter and  starting of the operation and list the file or at lest number of them and first five that will be affected.
  9. Verify the list of files you delete by running find or ls command first on wildcard you created.  When deleting  large number  of files first generate the list of files only only then apply rm command to this list.

Classic blunders with RM command

There are several classic blunders committed using the rm command :

  1. Running rm - R command without testing it using ls . You should always use ls -Rl command to test complex rm -R  commands (  -R, --recursive means process  subdirectories recursively).
  2. Mistyping complex path or file name in rm commands. It's always better to use copy and paste operation for directories and files used in rm command as it helps to avoid various typos. If you do recursive deletion is useful to have a "dry run" using find or ls to see the files.

    Here are to example of typos:

    rm -rf /tmp/foo/bar/ *

    instead of

    rm -rf /tmp/foo/bar/*

    ===

    rm -r /etc

    instead of

    rm -r etc

    This actually is an interesting type of error because /etc is typed daily so often that it kind of ingrained in sysadmin head and can be typed subconsciously instead of etc

  3. Using .*   That essentially make rm recursive as it matches ".." (parent directory).
  4. You can accidentally hit Enter by mistake before finishing typing the line containing rm command. 

Those cases can be viewed as shortcomings of rm implementation (it should automatically block * deletion of system directories like //etc/ and so on unless -f  flag is specified. As well as any system file.  Also Unix does not have system attribute for files although sticky bit on files can be used instead along with ownership of sys instead of root).

Writing a wrapper like "saferm"

Unix lacks system attributes for files although sticky bit on files or immutable attribute can be used instead.  It is wise to use wrappers for rm. There are several more or less usable approaches for writing such a wrapper:

In view of  danger of  rm -r unleashed as root it is wise to use wrappers for rm and alias rm to this wrapper in root .bash_profile or other profile file that you are using for root.  Aliases are not used in non-interactive sessions. So it will not affect any scripts. And in command line sessions rm is usually typed without path (which creates another set of dangers). 

There are several more or less usable features that might wish to experiment with when writing such a wrapper:

Of course each environment is unique and such a wrapper should take into account idiosyncrasies of such an environment.

If you know Perl, you can use Safe-rm as the initial implementation and enhance it. If it licensed under GPL. It present in Ubuntu I think.

How to use
-----------

Once you have installed safe-rm on your system (see INSTALL), you will need to
fill the system-wide or user-specific blacklists with the paths that you'd like
to protect against accidental deletion.

The system-wide blacklist lives in /etc/safe-rm.conf and you should probably add
paths like these:

  /
  /etc
  /usr
  /usr/lib
  /var

The user-specific blacklist lives in ~/.config/safe-rm and could include things like:

  /home/username/documents
  /home/username/documents/*
  /home/username/.mozilla


Other approaches
-----------------

If you want more protection than what safe-rm can offer, here are a few suggestions.

You could of course request confirmation every time you delete a file by putting this in
your /etc/bash.bashrc:

  alias rm='rm -i'

But this won't protect you from getting used to always saying yes, or from accidently
using 'rm -rf'.

Or you could make use of the Linux filesystem "immutable" attribute by marking (as root)
each file you want to protect:

  chattr +i file

Of course this is only usable on filesystems which support this feature.

Here are two projects which allow you to recover recently deleted files by trapping
all unlink(), rename() and open() system calls through the LD_PRELOAD facility:

  delsafe
  http://homepage.esoterica.pt/~nx0yew/delsafe/

  libtrashcan
  http://hpux.connect.org.uk/hppd/hpux/Development/Libraries/libtrash-0.2/readme.html

There are also projects which implement the FreeDesktop.org trashcan spec. For example:

  trash-cli
  http://code.google.com/p/trash-cli/

Finally, this project is a fork of GNU coreutils and adds features similar to safe-rm
to the rm command directly:

  http://wiki.github.com/d5h/rmfd/

Backup files before executing rm command, if you deleting a directory and it is not that large

Backups before execution of rm command are important. For example making backup of /etc directory on modern server takes a couple of seconds, but can save quite a lot of nerves in situations that otherwise can s can be devastating. For example, in the example above you would erase all your files and subdirectories in the /etc directory. Modern flavors of Unix usually prevent erasing / but not /etc. Linux which uses GNU version of rm prevents erasing all level 2 system directories.

You can also move directories to /Trash folder and delete them after, say 7 days or so.

You can also move the directory to /Trash folder is it is relatively small. In this forlder you can delete files that are say, older then 7 days with a cron script, if the total size of this directory exceeds certain threshold. 

More on blunders with the deleting a directory from the backup

I once automatically typed /etc instead of etc trying to delete directory to free space on a backup directory on a production server (/etc probably in engraved in sysadmin head as it is typed so often and can be substituted for etc subconsciously).  I realized that it was mistake and cancelled the command, but it was a fast server and one third of /etc was gone. The rest of the day was spoiled...  Actually not completely: I learned quite a bit about the behavior of AIX in this situation and the structure of AIX /etc directory this day so each such disaster is actually a great learning experience, almost like one day training course ;-). But it's much less nerve wracking to get this knowledge from the course... Another interesting thing is having backup was not enough is this case -- enterprise backup software stopped working on a damaged server. The same was true for telnet and ssh. And this was a remote server is a datacenter across the country.  I restored the directory on the other non-production server (overwriting its /etc directory in this second box with the help of operations, tell me about cascading errors and Murphy law :-). Then netcat helped to transfer the tar file. 

If you are working as root and perform dangerous operations never type a path of the command, copy it from the screen. If you can copy command from history instead of typing, do it !

Such blunders are really persistent as often used directories are types in a "semiconscious" fashion, in "autopilot" mode and you realize the blunder only after you hit Enter. For example, many years after the first blunder, I similarly typed rm -r /Soft instead of rm -r Soft in a backup directory.  And I was in a hurry so I did not even bother to load my profile and word on "bare root" account which did not have this "saferm" feature I was talking about earlier. Unfortunately for me /Soft was a huge directory so backup was not practical as I needed to free space.  It with all kind of software in it and the server was very fast to in a couple of seconds that elapsed before I realized the blunder approximately half gigabyte of files and directories was wiped out.  Luckily I have previous day backup.

In such cases network services with authentication stop working and the only way to transfer files is using CD/DVD, USB drive or netcat. That's why it is useful to have netcat on servers: netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-).

The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-). Please read classic sysadmin horror stories. 

A simple extra space often produced horrible consequences:

cd /usr/lib
ls /tmp/foo/bar

I typed
rm -rf /tmp/foo/bar/ *

instead of

rm -rf /tmp/foo/bar/*

The system doesn’t run very will without all of it’s libraries……

You can block such behavior requiring that if -r option was given you should have one and only one argument to rm. That should be a part of functionality of your "saferm" script.

Important class of subtle Unix errors: dot-star errors

Another poplar class of recursive rm errors are so called dot-star-errors, which often happn when rm is used with find.  Novice sysadmins usually do not realize that '.*' also matches '..' often with disastrous consequences.  If you are in any doubt as for how a wildcard will expand use echo command to see the result.

Using "convenience" symlinks to other (typically deeply nested) directories inside a directories and forgetting about them

Command  rm follows symlinks. So if you have a symlink to some system on important directory from the directory you are deleting all files in them will be deleted.

That's the main danger of "convenience symlinks" which are used to simplify access to deeply nested directories.  They are more dangerous then useful. Using aliases or functions is a better deal. If you use them, do this only from the level two directory created specially for this purpose like /Fav and protect this directory in your saferm script.

Disasterious typos in name of the directory or regex -- unintended space

There are some exotic typos that can lean you to troubles, especially with -r  option. One of them is unintended space:

 rm /homejoeuser/old*

instead of

rm /homejoeuser/old*

Or, similarly:

 rm * _old

instead of

rm *old
In all cases when you are deleting the directory directly copying its name from the listing obtained via ls  or find command is much safer that typing it by yourself.

Such a mistake is more damaging  is you use -r option. For example:

rm -rf /tmp/foo/bar/ *

instead of

rm -rf /tmp/foo/bar/*

So it is prudent to block execution of rm commands with multiple argument, if option -r is used in your saferm script.

Root deletion protection

To remove a file you must have write permission on the folder where it is stored. Sun introduced "rm -rf /" protection in Solaris 10, first released in 2005. Upon executing the command, the system now reports that the removal of / is not allowed.

Shortly after, the same functionality was introduced into FreeBSD version of rm  utility.

GNU rm  refuses to execute rm -rf /  if the --preserve-root  option is given, which has been the default since version 6.4 of GNU Core Utilities was released in 2006.

No such protection exists for critical system directories like /etc /bin, and so on but you can imitate it putting file "-i" into such directories or using a wrapper for the interactive usage of the command.  See also Safe-rm

Using echo command to see expansion of your wildcards, if any

To understand what will be actually being executed after shell expansion, preface your rm command with echo. For example if there is a filename starting with the dash you will receive very confusing message from rm

	$ rm *
	rm: unknown option -- -
	usage: rm [-f|-i] [-dPRrvW] file ...
To find out what caused it prefix the command with echo
echo rm *

You can generate list of files dynamically for rm (piping it via xarg if it is too large. For example

ls | grep copy # check what files will be affected

ls | grep copy | xargs -0 -l rm -f  # remove all the files

The simplest attempt to defend themselves against accidentally deleting files is to create an alias rm or function rm along the lines of:

alias rm="rm -i"

or use the function

function rm { /bin/rm -i "$@" ; }

Unfortunately, this encourage a tendency to mindlessly pound y  and the return key to affirm removes - until you realize that got just past the file you need to keep. Therefore it is better to create a function which first displays the files and then delete them:

function rm 
{ 
   if [ -z "$PS1" ] ; then
     /bin/rm "$@"
     exit $?
   fi 
   ls -l "$@"
   echo 'remove[ny]? ' | tr -d '\012' ;  
   read # read reply (y/n) 
   if [ "_$REPLY" = "_y" ]; then
      /bin/rm "$@" 
      exit $?   
   fi
   echo '(rm command cancelled)'
   exit 1 
} 

If this function is included into startup dot files it will be found ahead of the system  binary /bin/rm  in the search path and can break sloppy written batch jobs. That's why it checks for PS1 being non empty but this is not a 100% reliable test.

Using file starting with - as stoppers for unintended use of rm. Undeletable files

The rm command accepts the -- option which will cause it to stop processing flag options from that point forward. This allows the removal of file names that begin with a dash (-).

rm -- -filename

Actually accidental creation of files starting with dash is a common problem for novice sysadmin.  The problem is that if I type rm -foo, the rm command treats the filename as an option.

There are two simple ways to delete such a file. One was shown above and based on supplying an empty option argument. The second to use a relative or absolute pathname:

rm ./-badfile
 rm /home/user/-badfile 

Here the debugging trick that we mentioned above -- prefacing rm command with echo really helps to understand what happens.

To delete a file with non-printable characters in the name you can also use the shell wildcard "?" for each bad character

rm bad?file?name

Another way to delete files that have control characters in the name is to  use  -i option and an asterisk, which gives you the option of removing each file in the directory — even the ones that you can't type.

% rm -i *
rm: remove faq.html (y/n)? n
rm: remove foo (y/n)? y
%

The -i option may also be helpful when you are dealing with files with Unicode characters that appear to be regular letters in your locale, but that don't match patterns or names you use otherwise.

A great way to discover files with control characters in them is to use the -q option to the Unix ls command (some systems also support a useful -b option). You can, for example, alias the ls command:

alias lsq='ls -q '

Files that have control characters in their filenames will then appear with question marks:

ls -q f*
faq.html                fo?o

Blunders in using rm with find

This is class or blunders that deserves a page of its own. See Typical Errors In Using Find

You need to be extra careful running rm in an -exec option of find command. There are many horror stories in which people delete large parts of their filesystem due to typos of sloppy find command parameters. Again it is prudent first to test it using ls command to see the actual list of files to be deleted. Sometimes it is quite different from what you intended to do :-).

I often first generate the list using ls command, inspect it and in a separate step pipe it into rm :

cat rm.lst | xargs -t -n1 rm 
The -t flag causes the xargs command to display each command before running it, so you can see what is happening.

Dr. Nikolai Bezroukov

 


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Oct 22, 2018] linux - If I rm -rf a symlink will the data the link points to get erased, to

Notable quotes:
"... Put it in another words, those symlink-files will be deleted. The files they "point"/"link" to will not be touch. ..."
Oct 22, 2018 | unix.stackexchange.com

user4951 ,Jan 25, 2013 at 2:40

This is the contents of the /home3 directory on my system:
./   backup/    hearsttr@  lost+found/  randomvi@  sexsmovi@
../  freemark@  investgr@  nudenude@    romanced@  wallpape@

I want to clean this up but I am worried because of the symlinks, which point to another drive.

If I say rm -rf /home3 will it delete the other drive?

John Sui

rm -rf /home3 will delete all files and directory within home3 and home3 itself, which include symlink files, but will not "follow"(de-reference) those symlink.

Put it in another words, those symlink-files will be deleted. The files they "point"/"link" to will not be touch.

[Oct 22, 2018] Does rm -rf follow symbolic links?

Jan 25, 2012 | superuser.com
I have a directory like this:
$ ls -l
total 899166
drwxr-xr-x 12 me scicomp       324 Jan 24 13:47 data
-rw-r--r--  1 me scicomp     84188 Jan 24 13:47 lod-thin-1.000000-0.010000-0.030000.rda
drwxr-xr-x  2 me scicomp       808 Jan 24 13:47 log
lrwxrwxrwx  1 me scicomp        17 Jan 25 09:41 msg -> /home/me/msg

And I want to remove it using rm -r .

However I'm scared rm -r will follow the symlink and delete everything in that directory (which is very bad).

I can't find anything about this in the man pages. What would be the exact behavior of running rm -rf from a directory above this one?

LordDoskias Jan 25 '12 at 16:43, Jan 25, 2012 at 16:43

How hard it is to create a dummy dir with a symlink pointing to a dummy file and execute the scenario? Then you will know for sure how it works! –

hakre ,Feb 4, 2015 at 13:09

X-Ref: If I rm -rf a symlink will the data the link points to get erased, too? ; Deleting a folder that contains symlinkshakre Feb 4 '15 at 13:09

Susam Pal ,Jan 25, 2012 at 16:47

Example 1: Deleting a directory containing a soft link to another directory.
susam@nifty:~/so$ mkdir foo bar
susam@nifty:~/so$ touch bar/a.txt
susam@nifty:~/so$ ln -s /home/susam/so/bar/ foo/baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── foo
    └── baz -> /home/susam/so/bar/

3 directories, 1 file
susam@nifty:~/so$ rm -r foo
susam@nifty:~/so$ tree
.
└── bar
    └── a.txt

1 directory, 1 file
susam@nifty:~/so$

So, we see that the target of the soft-link survives.

Example 2: Deleting a soft link to a directory

susam@nifty:~/so$ ln -s /home/susam/so/bar baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── baz -> /home/susam/so/bar

2 directories, 1 file
susam@nifty:~/so$ rm -r baz
susam@nifty:~/so$ tree
.
└── bar
    └── a.txt

1 directory, 1 file
susam@nifty:~/so$

Only, the soft link is deleted. The target of the soft-link survives.

Example 3: Attempting to delete the target of a soft-link

susam@nifty:~/so$ ln -s /home/susam/so/bar baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── baz -> /home/susam/so/bar

2 directories, 1 file
susam@nifty:~/so$ rm -r baz/
rm: cannot remove 'baz/': Not a directory
susam@nifty:~/so$ tree
.
├── bar
└── baz -> /home/susam/so/bar

2 directories, 0 files

The file in the target of the symbolic link does not survive.

The above experiments were done on a Debian GNU/Linux 9.0 (stretch) system.

Wyrmwood ,Oct 30, 2014 at 20:36

rm -rf baz/* will remove the contents – Wyrmwood Oct 30 '14 at 20:36

Buttle Butkus ,Jan 12, 2016 at 0:35

Yes, if you do rm -rf [symlink], then the contents of the original directory will be obliterated! Be very careful. – Buttle Butkus Jan 12 '16 at 0:35

frnknstn ,Sep 11, 2017 at 10:22

Your example 3 is incorrect! On each system I have tried, the file a.txt will be removed in that scenario. – frnknstn Sep 11 '17 at 10:22

Susam Pal ,Sep 11, 2017 at 15:20

@frnknstn You are right. I see the same behaviour you mention on my latest Debian system. I don't remember on which version of Debian I performed the earlier experiments. In my earlier experiments on an older version of Debian, either a.txt must have survived in the third example or I must have made an error in my experiment. I have updated the answer with the current behaviour I observe on Debian 9 and this behaviour is consistent with what you mention. – Susam Pal Sep 11 '17 at 15:20

Ken Simon ,Jan 25, 2012 at 16:43

Your /home/me/msg directory will be safe if you rm -rf the directory from which you ran ls. Only the symlink itself will be removed, not the directory it points to.

The only thing I would be cautious of, would be if you called something like "rm -rf msg/" (with the trailing slash.) Do not do that because it will remove the directory that msg points to, rather than the msg symlink itself.

> ,Jan 25, 2012 at 16:54

"The only thing I would be cautious of, would be if you called something like "rm -rf msg/" (with the trailing slash.) Do not do that because it will remove the directory that msg points to, rather than the msg symlink itself." - I don't find this to be true. See the third example in my response below. – Susam Pal Jan 25 '12 at 16:54

Andrew Crabb ,Nov 26, 2013 at 21:52

I get the same result as @Susam ('rm -r symlink/' does not delete the target of symlink), which I am pleased about as it would be a very easy mistake to make. – Andrew Crabb Nov 26 '13 at 21:52

,

rm should remove files and directories. If the file is symbolic link, link is removed, not the target. It will not interpret a symbolic link. For example what should be the behavior when deleting 'broken links'- rm exits with 0 not with non-zero to indicate failure

[Oct 05, 2018] Sometimes one extra space makes a big differenece

Oct 05, 2018 | cam.ac.uk

From: rheiger@renext.open.ch (Richard H. E. Eiger)
Organization: Olivetti (Schweiz) AG, Branch Office Berne

In article <1992Oct9.100444.27928@u.washington.edu> tzs@stein.u.washington.edu
(Tim Smith) writes:
> I was working on a line printer spooler, which lived in /etc. I wanted
> to remove it, and so issued the command "rm /etc/lpspl." There was only
> one problem. Out of habit, I typed "passwd" after "/etc/" and removed
> the password file. Oops.
>
[deleted to save space[
>
> --Tim Smith

Here's another story. Just imagine having the sendmail.cf file in /etc. Now, I
was working on the sendmail stuff and had come up with lots of sendmail.cf.xxx
which I wanted to get rid of so I typed "rm -f sendmail.cf. *". At first I was
surprised about how much time it took to remove some 10 files or so. Hitting
the interrupt key, when I finally saw what had happened was way to late,
though.

Fortune has it that I'm a very lazy person. That's why I never bothered to just
back up directories with data that changes often. Therefore I managed to
restore /etc successfully before rebooting... :-) Happy end, after all. Of
course I had lost the only well working version of my sendmail.cf...

Richard

[Oct 05, 2018] Deletion of files purpose of which you do not understand sometimes backfire by Anatoly Ivasyuk

Oct 05, 2018 | cam.ac.uk

Unix Admin. Horror Story Summary, version 1.0 by Anatoly Ivasyuk

From: philip@haas.berkeley.edu (Philip Enteles)
Organization: Haas School of Business, Berkeley

As a new system administrator of a Unix machine with limited space I thought I was doing myself a favor by keeping things neat and clean. One
day as I was 'cleaning up' I removed a file called 'bzero'.

Strange things started to happen like vi didn't work then the compliants started coming in. Mail didn't work. The compilers didn't work. About this time the REAL system administrator poked his head in and asked what I had done.

Further examination showed that bzero is the zeroed memory without which the OS had no operating space so anything using temporary memory was non-functional.

The repair? Well things are tough to do when most of the utilities don't work. Eventually the REAL system administrator took the system to single user and rebuilt the system including full restores from a tape system. The Moral is don't be to anal about things you don't understand.

Take the time learn what those strange files are before removing them and screwing yourself.

Philip Enteles

[Oct 05, 2018] Danger of hidden symlinks

Oct 05, 2018 | cam.ac.uk

From: cjc@ulysses.att.com (Chris Calabrese)
Organization: AT&T Bell Labs, Murray Hill, NJ, USA

In article <7515@blue.cis.pitt.edu.UUCP> broadley@neurocog.lrdc.pitt.edu writes:
>On a old decstation 3100

I was deleting last semesters users to try to dig up some disk space, I also deleted some test users at the same time.

One user took longer then usual, so I hit control-c and tried ls. "ls: command not found"

Turns out that the test user had / as the home directory and the remove user script in Ultrix just happily blew away the whole disk.

>U...~

[Oct 05, 2018] Hidden symlinks and recursive deletion of the directories

Notable quotes:
"... Fucking asshole ex-sysadmin taught me a good lesson about checking for symlink bombs. ..."
Oct 05, 2018 | www.reddit.com

mavantix Jack of All Trades, Master of Some; 5 years ago (4 children)

I was cleaning up old temp folders of junk on Windows 2003 server, and C:\temp was full of shit. Most of it junk. Rooted deep in the junk, some asshole admin had apparently symlink'd sysvol to a folder in there. Deleting wiped sysvol.

There where no usable backups, well, there where but the ArcServe was screwed by lack of maintenance.

Spent days rebuilding policies.

Fucking asshole ex-sysadmin taught me a good lesson about checking for symlink bombs.

...and no I didn't tell this story to teach any of your little princesses to do the same when you leave your company.

[Oct 05, 2018] Automaticallt putting slash in from of directory named like system (named like bin,etc,usr, var) which is etched in sysadmin memory

This is why you should never type rm command on command line. Type it in editor first.
Oct 05, 2018 | www.reddit.com

aultl Senior DevOps Engineer

rm -rf /var

I was trying to delete /var/named/var

nekoeth0 Linux Admin, 5 years ago
Haha, that happened to me too. I had to use a live distro, chroot, copy, what not. It was fun!

[Oct 02, 2018] Rookie almost wipes customer's entire inventory unbeknownst to sysadmin

Notable quotes:
"... At that moment, everything from / and onward began deleting forcefully and Reginald described his subsequent actions as being akin to "flying flat like a dart in the air, arms stretched out, pointy finger fully extended" towards the power switch on the mini computer. ..."
Oct 02, 2018 | theregister.co.uk

I was going to type rm -rf /*.old* – which would have forcibly removed all /dgux.old stuff, including any sub-directories I may have created with that name," he said.

But – as regular readers will no doubt have guessed – he didn't.

"I fat fingered and typed rm -rf /* – and then I accidentally hit enter instead of the "." key."

At that moment, everything from / and onward began deleting forcefully and Reginald described his subsequent actions as being akin to "flying flat like a dart in the air, arms stretched out, pointy finger fully extended" towards the power switch on the mini computer.

"Everything got quiet."

Reginald tried to boot up the system, but it wouldn't. So instead he booted up off a tape drive to run the mini Unix installer and mounted the boot "/" file system as if he were upgrading – and then checked out the damage.

"Everything down to /dev was deleted, but I was so relieved I hadn't deleted the customer's database and only system files."

Reginald did what all the best accident-prone people do – kept the cock-up to himself, hoped no one would notice and started covering his tracks, by recreating all the system files.

Over the next three hours, he "painstakingly recreated the entire device tree by hand", at which point he could boot the machine properly – "and even the application worked out".

Jubilant at having managed the task, Reginald tried to keep a lid on the heart that was no doubt in his throat by this point and closed off his work, said goodbye to the sysadmin and went home to calm down. Luckily no one was any the wiser.

"If the admins read this message, this would be the first time they hear about it," he said.

"At the time they didn't come in to check what I was doing, and the system was inaccessible to the users due to planned maintenance anyway."

Did you feel the urge to confess to errors no one else at your work knew about? Do you know someone who kept something under their hat for years? Spill the beans to Who, Me? by emailing us here . ® Re: If rm -rf /* doesn't delete anything valuable

Eh? As I read it, Reginald kicked off the rm -rf /*, then hit the power switch before it deleted too much. The tape rescue revealed that "everything down to /dev" had been deleted, ie. everything in / beginnind a,b,c and some d. On a modern system that might include /boot and /bin, but evidently was not a total disaster on Reg's server.


Anonymous Coward

title="Inappropriate post? Report it to our moderators" type="submit" value="Report abuse"> I remember discovering the hard way that when you delete an email account in Thunderbird and it asks if you want to delete all the files associated with it it actually means do you want to delete the entire directory tree below where the account is stored .... so, as I discovered, saying "yes" when the reason you are deleting the account is because you'd just created it in the wrong place in the the directory tree is not a good idea - instead of just deleting the new account I nuked all the data associated with all our family email accounts!

big_D Monday 1st October 2018 10:05 GMT bpfh
div Re: .cobol

"Delete is right above Rename in the bloody menu"

Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

I found this out - to my peril - about 3 seconds before I realised that it was a good idea for a server's crontab to include a daily executed crontab -l > /foo/bar/crontab-backup.txt ...

Jason Bloomberg
div Re: .cobol

I went to delete the original files, but I only got as far as "del *.COB" befiore hitting return.

I managed a similar thing but more deliberately; belatedly finding "DEL FOOBAR.???" included files with no extensions when it didn't on a previous version (Win3.1?).

That wasn't the disaster it could have been but I've had my share of all-nighters making it look like I hadn't accidentally scrubbed a system clean.

Down not across
div Re: .cobol

Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

Using crontab -e is asking for trouble even without mistypes. I've see too many corrupted or truncated crontabs after someone has edited them with crontab -e. crontab -l > crontab.txt;vi crontab.txt;crontab crontab.txt is much better way.

You mean not everyone has crontab entry that backs up crontab at least daily?

MrBanana
div Re: .cobol

"WAH! I copied the .COBOL back to .COB and started over again. As I knew what I wanted to do this time, it only took about a day to re-do what I had deleted."

When this has happened to me, I end up with better code than I had before. Re-doing the work gives you a better perspective. Even if functionally no different it will be cleaner, well commented, and laid out more consistently. I sometimes now do it deliberately (although just saving the first new version, not deleting it) to clean up the code.

big_D
div Re: .cobol

I totally agree, the resultant code was better than what I had previously written, because some of the mistakes and assumptions I'd made the first time round and worked around didn't make it into the new code.

Woza
div Reminds me of the classic

https://www.ee.ryerson.ca/~elf/hack/recovery.html

Anonymous South African Coward
div Re: Reminds me of the classic

https://www.ee.ryerson.ca/~elf/hack/recovery.html

Was about to post the same. It is a legendary classic by now.

Chairman of the Bored
div One simple trick...

...depending on your shell and its configuration a zero size file in each directory you care about called '-i' will force the rampaging recursive rm, mv, or whatever back into interactive mode. By and large it won't defend you against mistakes in a script, but its definitely saved me from myself when running an interactive shell.

It's proven useful enough to earn its own cronjob that runs once a week and features a 'find -type d' and touch '-i' combo on systems I like.

Glad the OP's mad dive for the power switch saved him, I wasn't so speedy once. Total bustification. Hence this one simple trick...

Now if I could ever fdisk the right f$cking disk, I'd be set!

PickledAardvark
div Re: One simple trick...

"Can't you enter a command to abort the wipe?"

Maybe. But you still have to work out what got deleted.

On the first Unix system I used, an admin configured the rm command with a system alias so that rm required a confirmation. Annoying after a while but handy when learning.

When you are reconfiguring a system, delete/rm is not the only option. Move/mv protects you from your errors. If the OS has no move/mv, then copy, verify before delete.

Doctor Syntax
div Re: One simple trick...

"Move/mv protects you from your errors."

Not entirely. I had a similar experience with mv. I was left with a running shell so could cd through the remains of the file system end list files with echo * but not repair it..

Although we had the CDs (SCO) to reboot the system required a specific driver which wasn't included on the CDs and hadn't been provided by the vendor. It took most of a day before they emailed the correct driver to put on a floppy before I could reboot. After that it only took a few minutes to put everything back in place.

Chairman of the Bored
div Re: One simple trick...

@Chris Evans,

Yes there are a number of things you can do. Just like Windows a quick ctrl-C will abort a rm operation taking place in an interactive shell. Destroying the window in which the interactive shell running rm is running will work, too (alt-f4 in most window managers or 'x' out of the window)

If you know the process id of the rm process you can 'kill $pid' or do a 'killall -KILL rm'

Couple of problems:

(1) law of maximum perversity says that the most important bits will be destroyed first in any accident sequence

(2) by the time you realize the mistake there is no time to kill rm before law 1 is satisfied

The OP's mad dive for the power button is probably the very best move... provided you are right there at the console. And provided the big red switch is actually connected to anything

Colin Bull 1
div cp can also be dangerous

After several years working in a DOS environment I got a job as project Manager / Sys admin on a Unix based customer site for a six month stint. On my second day I wanted to use a test system to learn the software more, so decided to copy the live order files to the test system.

Unfortunately I forgot the trailing full stop as it was not needed in DOS - so the live order index file over wrote the live data file. And the company only took orders for next day delivery so it wiped all current orders.

Luckily it printed a sales acknowledgement every time an order was placed so I escaped death and learned never miss the second parameter with cp command.

Anonymous Coward

title="Inappropriate post? Report it to our moderators" type="submit" value="Report abuse"> i'd written a script to deploy the latest changes to the live environment. worked great. except one day i'd entered a typo and it was now deploying the same files to the remote directory, over and again.

it did that for 2 whole years with around 7 code releases. not a single person realised the production system was running the same code after each release with no change in functionality. all the customer cared about was 'was the site up?'

not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years??? Yet Another Anonymous coward

div Look on the bright side, any bugs your team had introduced in those 2 years had been blocked by your intrinsically secure script
Prst. V.Jeltz
div not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years???

That is Classic! not surprised about the AC!

Bet some of the beancounters were less than impressed , probly on customer side :)

Anonymous Coward

title="Inappropriate post? Report it to our moderators" type="submit" value="Report abuse"> Re: ...then there's backup stories...

Many years ago (pre internet times) a client phones at 5:30 Friday afternoon. It was the IT guy wanting to run through the steps involved in recovering from a backup. Their US headquarters had a hard disk fail on their accounting system. He was talking the Financial Controller through a recovery and while he knew his stuff he just wanted to double check everything.

8pm the same night the phone rang again - how soon could I fly to the states? Only one of the backup tapes was good. The financial controller had put the sole remaining good backup tape in the drive, then popped out to get a bite to eat at 7pm because it was going to be a late night. At 7:30pm the scheduled backup process copied the corrupted database over the only remaining backup.

Saturday was spent on the phone trying to talk them through everything I could think of.

Sunday afternoon I was sitting in a private jet winging it's way to their US HQ. Three days of very hard work later we'd managed to recreate the accounting database from pieces of corrupted databases and log files. Another private jet ride home - this time the pilot was kind enough to tell me there was a cooler full of beer behind my seat. Olivier2553

div Re: Welcome to the club!

"Lesson learned: NEVER decide to "clean up some old files" at 4:30 on a Friday afternoon. You WILL look for shortcuts and it WILL bite you on the ass."

Do not do anything of some significance on Friday. At all. Any major change, big operation, etc. must be made by Thursday at the latest, so in case of cock-up, you have the Friday (plus days week-end) to repair it.

JQW
div I once wiped a large portion of a hard drive after using find with exec rm -rf {} - due to not taking into account the fact that some directories on the system had spaces in them.
Will Godfrey
div Defensive typing

I've long been in the habit of entering dangerous commands partially in reverse, so in the case of theO/Ps one I've have done:

' -rf /*.old* '

then gone back top the start of the line and entered the ' rm ' bit.

sisk
div A couple months ago on my home computer (which has several Linux distros installed and which all share a common /home because I apparently like to make life difficult for myself - and yes, that's as close to a logical reason I have for having multiple distros installed on one machine) I was going to get rid of one of the extraneous Linux installs and use the space to expand the root partition for one of the other distros. I realized I'd typed /dev/sdc2 instead of /dev/sdc3 at the same time that I verified that, yes, I wanted to delete the partition. And sdc2 is where the above mentioned shared /home lives. Doh.

Fortunately I have a good file server and a cron job running rsync every night, so I didn't actually lose any data, but I think my heart stopped for a few seconds before I realized that.

Kevin Fairhurst
div Came in to work one Monday to find that the Unix system was borked... on investigation it appeared that a large number of files & folders had gone missing, probably by someone doing an incorrect rm.

Our systems were shared with our US office who supported the UK outside of our core hours (we were in from 7am to ensure trading was ready for 8am, they were available to field staff until 10pm UK time) so we suspected it was one of our US counterparts who had done it, but had no way to prove it.

Rather than try and fix anything, they'd gone through and deleted all logs and history entries so we could never find the evidence we needed!

Restoring the system from a recent backup brought everything back online again, as one would expect!

DavidRa
div Sure they did, but the universe invented better idiots

Of course. However, the incompletely-experienced often choose to force bypass that configuration. For example, a lot of systems aliased rm to "rm -i" by default, which would force interactive confirmations. People would then say "UGH, I hate having to do this" and add their own customisations to their shells/profiles etc:

unalias rm

alias rm=rm -f

Lo and behold, now no silly confirmations, regardless of stupidity/typos/etc.

[Apr 22, 2018] THE classic Unix horror story programming

Looks like not much changed since 1986. I amazed how little changed with Unix over the years. RM remains a danger although zsh and -I option on gnu rm are improvement. . I think every sysadmin wiped out important data with rm at least once in his career. So more work on this problem is needed.
Notable quotes:
"... Because we are creatures of habit. If you ALWAYS have to type 'yes' for every single deletion, it will become habitual, and you will start doing it without conscious thought. ..."
"... Amazing what kind of damage you can recover from given enough motivation. ..."
"... " in "rm -rf ~/ ..."
Apr 22, 2008 | www.reddit.com

probablycorey 10 years ago (35 children)

A little trick I use to ensure I never delete the root or home dir... Put a file called -i in / and ~

If you ever call rm -rf *, -i (the request confirmation option) will be the first path expanded. So your command becomes...

rm -rf -i

Catastrophe Averted!

mshade 10 years ago (0 children)
That's a pretty good trick! Unfortunately it doesn't work if you specify the path of course, but will keep you from doing it with a PWD of ~ or /.

Thanks!

aythun 10 years ago (2 children)
Or just use zsh. It's awesome in every possible way.
brian@xenon:~/tmp/test% rm -rf *
zsh: sure you want to delete all the files in /home/brian/tmp/test [yn]?
rex5249 10 years ago (1 child)
I keep an daily clone of my laptop and I usually do some backups in the middle of the day, so if I lose a disk it isn't a big deal other than the time wasted copying files.
MyrddinE 10 years ago (1 child)
Because we are creatures of habit. If you ALWAYS have to type 'yes' for every single deletion, it will become habitual, and you will start doing it without conscious thought.

Warnings must only pop up when there is actual danger, or you will become acclimated to, and cease to see, the warning.

This is exactly the problem with Windows Vista, and why so many people harbor such ill-will towards its 'security' system.

zakk 10 years ago (3 children)
and if I want to delete that file?!? ;-)
alanpost 10 years ago (0 children)
I use the same trick, so either of:

$ rm -- -i

or

$ rm ./-i

will work.

emag 10 years ago (0 children)
rm /-i ~/-i
nasorenga 10 years ago * (2 children)
The part that made me the most nostalgic was his email address: mcvax!ukc!man.cs.ux!miw

Gee whiz, those were the days... (Edit: typo)

floweryleatherboy 10 years ago (6 children)
One of my engineering managers wiped out an important server with rm -rf. Later it turned out he had a giant stock of kiddy porn on company servers.
monstermunch 10 years ago (16 children)
Whenever I use rm -rf, I always make sure to type the full path name in (never just use *) and put the -rf at the end, not after the rm. This means you don't have to worry about hitting "enter" in the middle of typing the path name (it won't delete the directory because the -rf is at the end) and you don't have to worry as much about data deletion from accidentally copy/pasting the command somewhere with middle click or if you redo the command while looking in your bash history.

Hmm, couldn't you alias "rm -rf" to mv the directory/files to a temp directory to be on the safe side?

branston 10 years ago (8 children)
Aliasing 'rm' is fairly common practice in some circles. It can have its own set of problems however (filling up partitions, running out of inodes...)
amnezia 10 years ago (5 children)
you could alias it with a script that prevents rm -rf * being run in certain directories.
jemminger 10 years ago (4 children)
you could also alias it to 'ls' :)
derefr 10 years ago * (1 child)
One could write a daemon that lets the oldest files in that directory be "garbage collected" when those conditions are approaching. I think this is, in a roundabout way, how Windows' shadow copy works.
branston 10 years ago (0 children)
Could do. Think we might be walking into the over-complexity trap however. The only time I've ever had an rm related disaster was when accidentally repeating an rm that was already in my command buffer. I looked at trying to exclude patterns from the command history but csh doesn't seem to support doing that so I gave up.

A decent solution just occurred to me when the underlying file system supports snapshots (UFS2 for example). Just snap the fs on which the to-be-deleted items are on prior to the delete. That needs barely any IO to do and you can set the snapshots to expire after 10 minutes.

Hmm... Might look at implementing that..

mbm 10 years ago (0 children)
Most of the original UNIX tools took the arguments in strict order, requiring that the options came first; you can even see this on some modern *BSD systems.
shadowsurge 10 years ago (1 child)
I just always format the command with ls first just to make sure everything is in working order. Then my neurosis kicks in and I do it again... and a couple more times just to make sure nothing bad happens.
Jonathan_the_Nerd 10 years ago (0 children)
If you're unsure about your wildcards, you can use echo to see exactly how the shell will expand your arguments.
splidge 10 years ago (0 children)
A better trick IMO is to use ls on the directory first.. then when you are sure that's what you meant type rm -rf !$ to delete it.
earthboundkid 10 years ago * (0 children)
Ever since I got burned by letting my pinky slip on the enter key years ago, I've been typing echo path first, then going back and adding the rm after the fact.
zerokey 10 years ago * (2 children)
Great story. Halfway through reading, I had a major wtf moment. I wasn't surprised by the use of a VAX, as my old department just retired their last VAX a year ago. The whole time, I'm thinking, "hello..mount the tape hardware on another system and, worst case scenario, boot from a live cd!"

Then I got to, "The next idea was to write a program to make a device descriptor for the tape deck" and looked back at the title and realized that it was from 1986 and realized, "oh..oh yeah...that's pretty fucked."

iluvatar 10 years ago (0 children)

Great story

Yeah, but really, he had way too much of a working system to qualify for true geek godhood. That title belongs to Al Viro . Even though I've read it several times, I'm still in awe every time I see that story...

cdesignproponentsist 10 years ago (0 children)
FreeBSD has backup statically-linked copies of essential system recovery tools in /rescue, just in case you toast /bin, /sbin, /lib, ld-elf.so.1, etc.

It won't protect against a rm -rf / though (and is not intended to), although you could chflags -R schg /rescue to make them immune to rm -rf.

clytle374 10 years ago * (9 children)
It happens, I tried a few months back to rm -rf bin to delete a directory and did a rm -rf /bin instead.

First thought: That took a long time.

Second thought: What do you mean ls not found.

I was amazed that the desktop survived for nearly an hour before crashing.

earthboundkid 10 years ago (8 children)
This really is a situation where GUIs are better than CLIs. There's nothing like the visual confirmation of seeing what you're obliterating to set your heart into the pit of your stomach.
jib 10 years ago (0 children)
If you're using a GUI, you probably already have that. If you're using a command line, use mv instead of rm.

In general, if you want the computer to do something, tell it what you want it to do, rather than telling it to do something you don't want and then complaining when it does what you say.

earthboundkid 10 years ago (3 children)
Yes, but trash cans aren't manly enough for vi and emacs users to take seriously. If it made sense and kept you from shooting yourself in the foot, it wouldn't be in the Unix tradition.
earthboundkid 10 years ago (1 child)
  1. Are you so low on disk space that it's important for your trash can to be empty at all times?
  2. Why should we humans have to adapt our directory names to route around the busted-ass-ness of our tools? The tools should be made to work with capital letters and spaces. Or better, use a GUI for deleting so that you don't have to worry about OMG, I forgot to put a slash in front of my space!

Seriously, I use the command line multiple times every day, but there are some tasks for which it is just not well suited compared to a GUI, and (bizarrely considering it's one thing the CLI is most used for) one of them is moving around and deleting files.

easytiger 10 years ago (0 children)
Thats a very simple bash/ksh/python/etc script.
  1. script a move op to a hidden dir on the /current/ partition.
  2. alias this to rm
  3. wrap rm as an alias to delete the contents of the hidden folder with confirmation
mattucf 10 years ago (3 children)
I'd like to think that most systems these days don't have / set as root's home directory, but I've seen a few that do. :/
dsfox 10 years ago (0 children)
This is a good approach in 1986. Today I would just pop in a bootable CDROM.
fjhqjv 10 years ago * (5 children)
That's why I always keep stringent file permissions and never act as the root user.

I'd have to try to rm -rf, get a permission denied error, then retype sudo rm -rf and then type in my password to ever have a mistake like that happen.

But I'm not a systems administrator, so maybe it's not the same thing.

toast_and_oj 10 years ago (2 children)
I aliased "rm -rf" to "omnomnom" and got myself into the habit of using that. I can barely type "omnomnom" when I really want to, let alone when I'm not really paying attention. It's saved one of my projects once already.
shen 10 years ago (0 children)
I've aliased "rm -rf" to "rmrf". Maybe I'm just a sucker for punishment.

I haven't been bit by it yet, the defining word being yet.

robreim 10 years ago (0 children)
I would have thought tab completion would have made omnomnom potentially easier to type than rm -rf (since the -rf part needs to be typed explicitly)
immure 10 years ago (0 children)
It's not.
lespea 10 years ago (0 children)
before I ever do something like that I make sure I don't have permissions so I get an error, then I press up, home, and type sudo <space> <enter> and it works as expected :)
kirun 10 years ago (0 children)
And I was pleased the other day how easy it was to fix the system after I accidentally removed kdm, konqueror and kdesktop... but these guys are hardcore.
austin_k 10 years ago (0 children)
I actually started to feel sick reading that. I've been in a IT disaster before where we almost lost a huge database. Ugh.. I still have nightmares.
umilmi81 10 years ago (4 children)
Task number 1 with a UNIX system. Alias rm to rm -i. Call the explicit path when you want to avoid the -i (ie: /bin/rm -f). Nobody is too cool to skip this basic protection.
flinchn 10 years ago (0 children)
i did an application install at an LE agency last fall - stupid me mv ./etc ./etcbk <> mv /etc /etcbk

ahh that damned period

DrunkenAsshole 10 years ago (0 children)
Were the "*"s really needed for a story that has plagued, at one point or another, all OS users?
xelfer 10 years ago (0 children)
Is the home directory for root / for some unix systems? i thought 'cd' then 'rm -rf *' would have deleted whatever's in his home directory (or whatever $HOME points to)
srparish 10 years ago (0 children)
Couldn't he just have used the editor to create the etc files he wanted, and used cpio as root to copy that over as an /etc?

sRp

stox 10 years ago (1 child)
Been there, done that. Have the soiled underwear to prove it. Amazing what kind of damage you can recover from given enough motivation.
sheepskin 10 years ago * (0 children)
I had a customer do this, he killed it about the same time. I told him he was screwed and I'd charge him a bunch of money to take down his server, rebuild it from a working one and put it back up. But the customer happened to have a root ftp session up, and was able to upload what he needed to bring the system back. by the time he was done I rebooted it to make sure it was cool and it booted all the way back up.

Of course I've also had a lot of customer that have done it, and they where screwed, and I got to charge them a bunch of money.

jemminger 10 years ago (0 children)
pfft. that's why lusers don't get root access.
supersan 10 years ago (2 children)
i had the same thing happened to me once.. my c:\ drive was running ntfs and i accidently deleted the "ntldr" system file in the c:\ root (because the name didn't figure much).. then later, i couldn't even boot in the safe mode! and my bootable disk didn't recognize the c:\ drive because it was ntfs!! so sadly, i had to reinstall everything :( wasted a whole day over it..
b100dian 10 years ago (0 children)
Yes, but that's a single file. I suppose anyone can write hex into mbr to copy ntldr from a samba share!
bobcat 10 years ago (0 children)
http://en.wikipedia.org/wiki/Emergency_Repair_Disk
boredzo 10 years ago (0 children)
Neither one is the original source. The original source is Usenet, and I can't find it with Google Groups. So either of these webpages is as good as the other.
docgnome 10 years ago (0 children)
In 1986? On a VAX?
MarlonBain 10 years ago (0 children)

This classic article from Mario Wolczko first appeared on Usenet in 1986 .

amoore 10 years ago (0 children)
I got sidetracked trying to figure out why the fictional antagonist would type the extra "/ " in "rm -rf ~/ ".
Zombine 10 years ago (2 children)

...it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal.

Yeah. So apart from the fact that no one could get any work done or really do anything, things were working great!

I think a more rational reaction would be "Why on Earth is this big, important system on which many people rely designed in such a way that a simple easy-to-make human error can screw it up so comprehensively?" or perhaps "Why on Earth don't we have a proper backup system?"

daniels220 10 years ago (1 child)
The problem wasn't the backup system, it was the restore system, which relied on the machine having a "copy" command. Perfectly reasonable assumption that happened not to be true.
Zombine 10 years ago * (0 children)
Neither backup nor restoration serves any purpose in isolation. Most people would group those operations together under the heading "backup;" certainly you win only a semantic victory by doing otherwise. Their fail-safe data-protection system, call it what you will, turned out not to work, and had to be re-engineered on-the-fly.

I generally figure that the assumptions I make that turn out to be entirely wrong were not "perfectly reasonable" assumptions in the first place. Call me a traditionalist.

[Apr 22, 2018] rm and Its Dangers (Unix Power Tools, 3rd Edition)

Apr 22, 2018 | docstore.mik.ua
14.3. rm and Its Dangers

Under Unix, you use the rm command to delete files. The command is simple enough; you just type rm followed by a list of files. If anything, rm is too simple. It's easy to delete more than you want, and once something is gone, it's permanently gone. There are a few hacks that make rm somewhat safer, and we'll get to those momentarily. But first, here's a quick look at some of the dangers.

To understand why it's impossible to reclaim deleted files, you need to know a bit about how the Unix filesystem works. The system contains a "free list," which is a list of disk blocks that aren't used. When you delete a file, its directory entry (which gives it its name) is removed. If there are no more links ( Section 10.3 ) to the file (i.e., if the file only had one name), its inode ( Section 14.2 ) is added to the list of free inodes, and its datablocks are added to the free list.

Well, why can't you get the file back from the free list? After all, there are DOS utilities that can reclaim deleted files by doing something similar. Remember, though, Unix is a multitasking operating system. Even if you think your system is a single-user system, there are a lot of things going on "behind your back": daemons are writing to log files, handling network connections, processing electronic mail, and so on. You could theoretically reclaim a file if you could "freeze" the filesystem the instant your file was deleted -- but that's not possible. With Unix, everything is always active. By the time you realize you made a mistake, your file's data blocks may well have been reused for something else.

When you're deleting files, it's important to use wildcards carefully. Simple typing errors can have disastrous consequences. Let's say you want to delete all your object ( .o ) files. You want to type:

% rm *.o

But because of a nervous twitch, you add an extra space and type:

% rm * .o

It looks right, and you might not even notice the error. But before you know it, all the files in the current directory will be gone, irretrievably.

If you don't think this can happen to you, here's something that actually did happen to me. At one point, when I was a relatively new Unix user, I was working on my company's business plan. The executives thought, so as to be "secure," that they'd set a business plan's permissions so you had to be root ( Section 1.18 ) to modify it. (A mistake in its own right, but that's another story.) I was using a terminal I wasn't familiar with and accidentally created a bunch of files with four control characters at the beginning of their name. To get rid of these, I typed (as root ):

# rm ????*

This command took a long time to execute. When about two-thirds of the directory was gone, I realized (with horror) what was happening: I was deleting all files with four or more characters in the filename.

The story got worse. They hadn't made a backup in about five months. (By the way, this article should give you plenty of reasons for making regular backups ( Section 38.3 ).) By the time I had restored the files I had deleted (a several-hour process in itself; this was on an ancient version of Unix with a horrible backup utility) and checked (by hand) all the files against our printed copy of the business plan, I had resolved to be very careful with my rm commands.

[Some shells have safeguards that work against Mike's first disastrous example -- but not the second one. Automatic safeguards like these can become a crutch, though . . . when you use another shell temporarily and don't have them, or when you type an expression like Mike's very destructive second example. I agree with his simple advice: check your rm commands carefully! -- JP ]

-- ML

[Apr 22, 2018] How to prevent a mistaken rm -rf for specific folders?

Notable quotes:
"... There's nothing more on a traditional Linux, but you can set Apparmor/SELinux/ rules that prevent rm from accessing certain directories. ..."
"... Probably your best bet with it would be to alias rm -ri into something memorable like kill_it_with_fire . This way whenever you feel like removing something, go ahead and kill it with fire. ..."
Jan 20, 2013 | unix.stackexchange.com

I think pretty much people here mistakenly 'rm -rf'ed the wrong directory, and hopefully it did not cause a huge damage.. Is there any way to prevent users from doing a similar unix horror story?? Someone mentioned (in the comments section of the previous link) that

... I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it ...

Is there any implementation of that in any current Unix or Linux distro? And what is the common practice to prevent that error even from a sysadmin (with root access)?

It seems that there was some protection for the root directory (/) in Solaris (since 2005) and GNU (since 2006). Is there anyway to implement the same protection way to some other folders as well??

To give it more clarity, I was not asking about general advice about rm usage (and I've updated the title to indicate that more), I want something more like the root folder protection: in order to rm -rf / you have to pass a specific parameter: rm -rf --no-preserve-root /.. Is there similar implementations for customized set of directories? Or can I specify files in addition to / to be protected by the preserve-root option?


amyassin, Jan 20, 2013 at 17:26

I think pretty much people here mistakenly ' rm -rf 'ed the wrong directory, and hopefully it did not cause a huge damage.. Is there any way to prevent users from doing a similar unix horror story ?? Someone mentioned (in the comments section of the previous link ) that

... I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it ...

Is there any implementation of that in any current Unix or Linux distro? And what is the common practice to prevent that error even from a sysadmin (with root access)?

It seems that there was some protection for the root directory ( / ) in Solaris (since 2005) and GNU (since 2006). Is there anyway to implement the same protection way to some other folders as well??

To give it more clarity, I was not asking about general advice about rm usage (and I've updated the title to indicate that more), I want something more like the root folder protection: in order to rm -rf / you have to pass a specific parameter: rm -rf --no-preserve-root / .. Is there similar implementations for customized set of directories? Or can I specify files in addition to / to be protected by the preserve-root option?

mattdm, Jan 20, 2013 at 17:33

1) Change management 2) Backups. – mattdm Jan 20 '13 at 17:33

Keith, Jan 20, 2013 at 17:40

probably the only way would be to replace the rm command with one that doesn't have that feature. – Keith Jan 20 '13 at 17:40

sr_, Jan 20, 2013 at 18:28

safe-rm maybe – sr_ Jan 20 '13 at 18:28

Bananguin, Jan 20, 2013 at 21:07

most distros do `alias rm='rm -i' which makes rm ask you if you are sure.

Besides that: know what you are doing. only become root if necessary. for any user with root privileges security of any kind must be implemented in and by the user. hire somebody if you can't do it yourself.over time any countermeasure becomes equivalaent to the alias line above if you cant wrap your own head around the problem. – Bananguin Jan 20 '13 at 21:07

midnightsteel, Jan 22, 2013 at 14:21

@amyassin using rm -rf can be a resume generating event. Check and triple check before executing it – midnightsteel Jan 22 '13 at 14:21

Gilles, Jan 22, 2013 at 0:18

To avoid a mistaken rm -rf, do not type rm -rf .

If you need to delete a directory tree, I recommend the following workflow:

Never call rm -rf with an argument other than DELETE . Doing the deletion in several stages gives you an opportunity to verify that you aren't deleting the wrong thing, either because of a typo (as in rm -rf /foo /bar instead of rm -rf /foo/bar ) or because of a braino (oops, no, I meant to delete foo.old and keep foo.new ).

If your problem is that you can't trust others not to type rm -rf, consider removing their admin privileges. There's a lot more that can go wrong than rm .


Always make backups .

Periodically verify that your backups are working and up-to-date.

Keep everything that can't be easily downloaded from somewhere under version control.


With a basic unix system, if you really want to make some directories undeletable by rm, replace (or better shadow) rm by a custom script that rejects certain arguments. Or by hg rm .

Some unix variants offer more possibilities.

amyassin, Jan 22, 2013 at 9:41

Yeah backing up is the most amazing solution, but I was thinking of something like the --no-preserve-root option, for other important folder.. And that apparently does not exist even as a practice... – amyassin Jan 22 '13 at 9:41

Gilles, Jan 22, 2013 at 20:32

@amyassin I'm afraid there's nothing more (at least not on Linux). rm -rf already means "delete this, yes I'm sure I know what I'm doing". If you want more, replace rm by a script that refuses to delete certain directories. – Gilles Jan 22 '13 at 20:32

Gilles, Jan 22, 2013 at 22:17

@amyassin Actually, I take this back. There's nothing more on a traditional Linux, but you can set Apparmor/SELinux/ rules that prevent rm from accessing certain directories. Also, since your question isn't only about Linux, I should have mentioned OSX, which has something a bit like what you want. – Gilles Jan 22 '13 at 22:17

qbi, Jan 22, 2013 at 21:29

If you are using rm * and the zsh, you can set the option rmstarwait :
setopt rmstarwait

Now the shell warns when you're using the * :

> zsh -f
> setopt rmstarwait
> touch a b c
> rm *
zsh: sure you want to delete all the files in /home/unixuser [yn]? _

When you reject it ( n ), nothing happens. Otherwise all files will be deleted.

Drake Clarris, Jan 22, 2013 at 14:11

EDIT as suggested by comment:

You can change the attribute of to immutable the file or directory and then it cannot be deleted even by root until the attribute is removed.

chattr +i /some/important/file

This also means that the file cannot be written to or changed in anyway, even by root . Another attribute apparently available that I haven't used myself is the append attribute ( chattr +a /some/important/file . Then the file can only be opened in append mode, meaning no deletion as well, but you can add to it (say a log file). This means you won't be able to edit it in vim for example, but you can do echo 'this adds a line' >> /some/important/file . Using > instead of >> will fail.

These attributes can be unset using a minus sign, i.e. chattr -i file

Otherwise, if this is not suitable, one thing I practice is to always ls /some/dir first, and then instead of retyping the command, press up arrow CTL-A, then delete the ls and type in my rm -rf if I need it. Not perfect, but by looking at the results of ls, you know before hand if it is what you wanted.

NlightNFotis, Jan 22, 2013 at 8:27

One possible choice is to stop using rm -rf and start using rm -ri . The extra i parameter there is to make sure that it asks if you are sure you want to delete the file.

Probably your best bet with it would be to alias rm -ri into something memorable like kill_it_with_fire . This way whenever you feel like removing something, go ahead and kill it with fire.

amyassin, Jan 22, 2013 at 14:24

I like the name, but isn't f is the exact opposite of i option?? I tried it and worked though... – amyassin Jan 22 '13 at 14:24

NlightNFotis, Jan 22, 2013 at 16:09

@amyassin Yes it is. For some strange kind of fashion, I thought I only had r in there. Just fixed it. – NlightNFotis Jan 22 '13 at 16:09

Silverrocker, Jan 22, 2013 at 14:46

To protect against an accidental rm -rf * in a directory, create a file called "-i" (you can do this with emacs or some other program) in that directory. The shell will try to interpret -i and will cause it to go into interactive mode.

For example: You have a directory called rmtest with the file named -i inside. If you try to rm everything inside the directory, rm will first get -i passed to it and will go into interactive mode. If you put such a file inside the directories you would like to have some protection on, it might help.

Note that this is ineffective against rm -rf rmtest .

ValeriRangelov, Dec 21, 2014 at 3:03

If you understand C programming language, I think it is possible to rewrite the rm source code and make a little patch for kernel. I saw this on one server and it was impossible to delete some important directories and when you type 'rm -rf /direcotyr' it send email to sysadmin.

bash - Safe Directory Delete

Mar 13, 2013 | Stack Exchange

rm -i would get extremely annoying especially when removing recursively.

rm -I, on the other hand seems a little more tolerable which will prompt once before removing more than three files, or when removing recursively.

It's much less intrusive while still giving protection against most mistakes.

The best form of protection is to double/triple check when deleting recursively or better yet, anything you delete.

You can cd to the top level directory you want to remove recursively, then go up a level and rm that directory. That way you don't have to use leading slashes, etc.

cd /home/user/junk
ls -1

folders
to
delete

rm -ri . ###Or "rm -rf ."
cd ..
rm -rf junk

Another way to check that your paths are right for any command is to simply use tab completion.

One option is to use a trash can approach, like here.

There are a lot of examples out there besides this one - it was just the first in my search results. Basically, all rm operations become mv commands. If you rm the wrong files, they're still on the machine and easily recoverable. And the approach linked above creates an alias to the base rm command so you can still use it when you really want to.

[Apr 22, 2018] bash - Is it safe to alias rm to safe-rm

Jun 7, 2014 | Ask Ubuntu

Q:

We are happy to use safe-rm as a replacement to rm, but is it considered safe to alias it?

e.g.

alias rm='safe-rm'
A: Yes, an alias is just a shell-only rename. The scope of an alias is only the shell that you are currently in. safe-rm is pDELETE. Doing the deletion in several stages gives you an opportunity to verify that you aren't deleting the wrong thing, either because of a typo (as in rm -rf /foo /bar instead of rm -rf /foo/bar) or because of a braino (oops, no, I meant to delete foo.old and keep foo.new).

If your problem is that you can't trust others not to type rm -rf, consider removing their admin privileges. There's a lot more that can go wrong than rm.

Let me get into the problem you asked

This tool, safe-rm, ships with a default system-wide configuration file (/etc/safe-rm.conf), but each user can supplement that list of "protected" directories and files by adding lines to ~/.safe-rm.

Use \rm when you are sure about what you are deleting but most of the time, I prefer to go through the confirmation. Too many files disappeared because of a rushed removal action.

THE classic Unix horror story WSU Linux Users Group

*THE* classic Unix horror story

Tue, 01/31/2006 - 10:11am - binford2k

Try this in windows!


Unix Recovery Legend

This classic article from Mario Wolczko first appeared on Usenet in 1986.

Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters....

It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg. My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed

grep 147 /etc/passwd

only to receive the response

/etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss. This was confirmed when in response to

ls /etc

I got

ls: not found.

I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager.

When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines:

# cd
# rm -rf *

Oh, shit, I thought. That would just about explain it.

I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and typing

cd /
echo *

I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know[work?] without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck.

Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable.

So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference:

070100002c000000000000000000000000000000000000000000000000000000
0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
8800040000bc012f65746300

I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved.

So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history.

Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one.

The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat.

And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together.

I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to:

Mario Wolczko
------------------------------------------------------------------------
Dept. of Computer Science       ARPA:   miw%uk.ac.man.cs.ux@cs.ucl.ac.uk
The University                  USENET: mcvax!ukc!man.cs.ux!miw
Manchester M13 9PL              JANET:  miw@uk.ac.man.cs.ux
U.K.                            061-273 7121 x 5699
------------------------------------------------------------------------
Re: *THE* classic Unix horror story

We used to outsource some of our support at our data center, and we had multiple rm's from those agents. Luckily, live CD's can get you back rolling to where you can rsync from backups.

# cd
# rm -rf *

If I'm not mistaken, wouldn't this just delete everything in /root ? I'm more of a Linux guy, but I'd presume you'd need the following command to wipe everything.

rm -rf /

-Chris | Editor
Mozy Reviews

Mon, 04/25/2011 - 3:43pm - binford2k
Re: *THE* classic Unix horror story

It all depends on where root's home dir is. Some unixes do have it set to /

Thu, 05/19/2011 - 9:36am - mchristenson
Re: *THE* classic Unix horror story

It doesn't matter whether you keep anything in /root or /home, if the distribution you are using has the root user's HOME environment variable set to /.

POSIX standard says that, when not supplied with any arguments, the cd command will change to the directory pointed to by HOME (or an implementation based location if HOME is unset or undefined.) If the former, subsequently invoking rm -rf * would then remove both your /root and /home directories.

See http://pubs.opengroup.org/onlinepubs/009695399/utilities/cd.html

Mon, 01/03/2011 - 12:39am - scott.allerdice
Re: *THE* classic Unix horror story

Not classic horrible yes, the admins did a good job of killing the process before it made it's way to the rest of the file system. I had a similar problem on windows in the 90's when malware didn't exist and viruses weren't bots but destroyed files. I went to start run telnet and it started searching for telnet.exe i knew something was wrong so went to my windows folder to look for it manually and didn't see it. How strange i thought i just saw a file disappear and another etc. Yup files were disappearing before my very eyes. It was as bad as rm -rf and i had no time to look and see what was doing it so i pulled the plug on my machine. It's what i would do had i seen rm -rf. Pull the power and figure out a plan. For windows i had norton on floppy and was able to clean it out i don't know what i lost but got the system back up and running. Unix wise i've ran redhat, slack, suse, ubuntu, freebsd, and sparc x86 i make it a point to have weekly backups of my drives and with out using recovery CD's or programs i do it almost manually copying files over to the drives so i know they are in a format i can access once recovered. If a friend ever threatened an rm -rf for $5 i would comply however his next trip home would be memorable i would fill his suite case with sex toys and synthetic marijuana and coke powder, it's legal and used to train drug sniffing dogs. It would make for a hell of a trip through TSA screening and possibly a trip to jail court or maybe even drug rehab i'd tell him he can add the $5 to the bail money i'd put up for him.

Re: *THE* classic Unix horror story

Heh, ingenious and creative. I would have given up about half way through (if that)

Re: *THE* classic Unix horror story

This is without a doubt the most convoluted restore process I've ever heard of. It sounds to me like the operators involved were too naïve when it came to the install process for 4.3BSD. It really isn't that involved. Honestly, you have to wonder who installed their OS, and why didn't they have access to the manuals? I mean even if they paid for the tape only, wouldn't they have printed out the system manuals?

I could have had that box up & running again in under an hour. First thing to do would be to locate the install tape, and halt the machine. Boot from tape, and run the copy program to install the miniroot into the swap partition. Rebooting the VAX again, into the swap partition I would restore the root partition from one week old root backup.

One more reboot, and the VAX would have it's root back from their best backup, then it would be a matter of restoring other directories as needed.

Oh and yes by default 4.3BSD has separate root partition from the rest of the userland stuff.

I can only think that these people were absolutely terrified of rebooting their VAX. I wonder if it was an 11/780 with a borked console tape/floppy.

But who knows this is over 20 years ago.

Why not use tar

One way to avoid the assembler is to use a combination of tar and uuencode

Basically, if you transfer a tar file with only one directory in it (by manual typing) - you would be able to extract it using

tar xvf etc.tar -C /

Only issue is that the size is a bit larger - but I think if one gives enough thought to this - it could be reduced.

The key to this ofcourse - as you say, is to NOT PANIC.

Fri, 03/14/2008 - 3:11pm - joeblow1470
Re: Why not use tar

tar is located at /bin/tar as it states in the story(about 1/3 of the way down)(bin was one of the deleted directories)

PS great story

Thu, 10/07/2010 - 10:51am - jfwfmt
Re: *THE* classic Unix Horror story

not on Unix, on RT-11 but a close equivalent horror story:

One night I was finishing up the code for the VAX 8600 console. I had finally made the changes to the OS code and to the clock driver to translate the 64Hz interrupt rate to the 60Hz clock on the PDP-11 (with a little studder every 1/4 second) and to transfer the 1KHz clock through the mismatched registers windows to the VAX.

So everything was good and i decided to clean up and leave. I entered:

. delete *,tmp

It took a long time and then came back with:

tmp file not found.

Of course the comma rather than the intended dot made all the difference.

. directory

returned with nothing.

Not to worry, there were men in white coats who religiously took backups of everything every night,

Except they had never done a restore.

Which of course failed.

So I came in the next day and started over.

Fri, 12/10/2010 - 10:00pm - greffedecheveux (not verified)

[Apr 21, 2018] Any alias of rm is a very stupid idea

Option -I is more modern and more useful then old option -i. It is highly recommended. And it make sense to to use alias with it contrary to what this author states (he probably does not understand that aliases do not wok for non-interactive sessions.).
The point the author make is that when you automatically expect rm to be aisles to rm -i you get into trouble on machines where this is not the case. And that's completely true.
But it does not solve the problem as respondents soon became automatic. stated. Writing your own wrapper is a better deal. One such wrapper -- safe-rm already exists and while not perfect is useful
Notable quotes:
"... A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do. ..."
"... It you want a safety net, do "alias del='rm -I –preserve_root'", ..."
Feb 14, 2017 | www.cyberciti.biz
Art Protin June 12, 2012, 9:53 pm

Any alias of rm is a very stupid idea (except maybe alias rm=echo fool).

A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do.

It you want a safety net, do "alias del='rm -I –preserve_root'",

Drew Hammond March 26, 2014, 7:41 pm
^ This x10000.

I've made the same mistake before and its horrible.

[Jun 12, 2010] Sysadmin Blunder (3rd Response)

Toolbox for IT Groups

jxtmills:

I needed to clear a directory full of ".directories" and I issued:

rm -r .*

It seemed to find ".." awfully fast. That was a bad day, but, we restored most of the files in about an hour.

John T. Mills

alain:

hi every one

for me , i remember 2

1 - 'rm *;old' in / directory note ';' instead of '.'

2 - kill #pid of the informix process (oninit) and delete it (i dreamed)

jturner:

Variation on a theme ... the 'rm -r theme'

as a junior admin on AIX3.2.5, I had been left to my own devices to create
some housekeeping scripts - all my draft scripts being created and tested
in my home directory with a '.jt' suffix. After completing the scripts I
found that I had inadvertantly placed some copies in / with a .jt suffix.
Easy job then to issue a 'rm *.jt' in / and all would be well. Well it
would have been if I hadn't put a space between the * and the .jt. And the
worst thing of all, not being a touch typist and looking at the keys, I
glanced at the screen before hitting the enter key and realised with horror
what was going to happen and STILL my little finger continued to proceed
toward the enter key.

Talk about 'Gone in 60 seconds' - my life was at an end - over - finished
- perhaps I could park cars or pump gas for a living. Like other
correspondents a helpful senior admin was on hand to smile kindly and show
me how to restore from mksysb-fortunately taken daily on these production
systems. (Thanks Pete :))) )

To this day, rm -i is my first choice with multiple rm's just as a test!!!!!!

Happy rm-ing :)

daguenet:

I know that one. Does any body remember when the rm man page had a warning not do rm -rf / as root? How may systems were rebuild due that blunder. Not that I have ever done something like that, nor will ever admit to it:).

Aaron

cal.staples:

That is a no brainer!

First a little background. I cooked up a script called "killme" which
would ask for a search string then parse the process table and return a
list of all matches. If the list contained the processes you wanted to
kill then you would answer "Yes", not once, but twice just to be sure
you thought about it. This was very handy at times so I put it out on
all of our servers.

Some time had passed and I had not used it for a while when I had a need
to kill a group of processes. So I typed the command not realizing that
I had forgotten the scripts name. Of course I was on our biggest
production system at that time and everything stopped in it's tracks!

Unknown to me was that there is an AIX command called "killall" which is
what I typed.

From the MAN page: "The killall command cancels all processes that you
started, except those producing the killall process. This command
provides a convenient means of canceling all processes created by the
shell that you control. When started by a root user, the killall command
cancels all cancellable processes except those processes that started
it."

And it doesn't ask for confirmation or anything! Fortunately the
database didn't get corrupted and we were able to bring everything back
on line fairly quickly. Needless to say we changed the name of this
command so it couldn't be run so easily.

"Killall" is a great command for a programmer developing an application
which goes wild and he/she needs to kill the processes and retain
control of the system, but it is very dangerous in the real world!

Jeff Scott:

The silliest mistake? That had to be a permissions change on /bin. I got a
call from an Oracle DBA that the $ORACLE_HOME/bin no longer belonged to
oracle:dba. We never found out how that happened. I logged in to change the
permissions. I accidentally typed cd /oracle.... /bin (note the space
before /bin), then cheerfully entered the following command:

#chown -R oracle:dba ./*

The command did not climb up to root fortunately, but it really made a mess.
We ended up restoring /bin from a backup taken the previous evening.

Jeff Scott
Darwin Partners/EMC

Cal S.

tzhou:

crontab -r when I wanted to do crontab -e. See letters e and r are side by
side on the keyboard. I had 2 pages of crontab and had no backup on the
machine !

Jeff Scott:

I've seen the rm -fr effect before. There were no entries in any crontab.
Before switching to sudo, the company used a homegrown utility to grant
things root access. The server accepted ETLs from other databases, acting as
a data warehouse. This utility logged locally, instead of logging via
syslogd with local and remote logging. So, when the system was erased, there
really was no way to determine the actual cause. Most of the interactive
users shared a set of group accounts, instead of enforcing individual
accounts and using su - or sudo. The outage cost the company $8 million USD
due to lost labor that had to be repeated. Clearly, it was caused by a
cleanup script, but it is anyone's guess which one.

Technically, this was not a sysadmin blunder, but it underscores the need
for individual accounts and for remote logging of ALL uses of group
accounts, including those performed by scripts.

It also underscores the absolute requirement that all scripts have error
trapping mechanisms. In this case, the rm -fr was likely preceded by a cd to
a nonexistent directory. Since this was performed as root, the script cd'd
to / instead. The rm -fr then removed everything. The other possibility is
that it applied itself to another directory, but, again, root privileges
allowed it to climb up the directory tree to root.

Creative uses of rm

See also Syaadmin Horror Stories

There's an interesting thread on the CentOS mailing list on sys admin horror stories. Amusing read.

http://lists.centos.org/pipermail/centos/2010-June/095423.html

[Ilugc] [OT] sys admin horror stories

I once had a vendor sys admin run an "rm -rf ./project/ subdir" and delete the entire ./project folder and 2 months of live data.

Same vendor had also not noticed that automated backups had failed for many weeks and we were running without any backup for a long time.

A penalty of Rs 2 lakhs was enforced, all because of one extra space in the command.

- Raja

Greatest blunders

IT Resource Center

Christian Gebhardt

Hi
As a newby in UNIX I had an Oracle Testinstallion on a production system
productiv directory: /u01/...
test directory: /test/u01/...

deleting the test installation:
cd /test
rm /u01

OOPS ...

After several bdf commands I noticed that the wrong lvol shrinks and stops the delete command with Ctrl'C

The database still worked without the most binaries and libraries and after a restore from tape without stopping and starting the database all was ok.

I love oracle ;-)

Chris

harry d brown jr

Learning hpux? Naw, that's not it....maybe it was learning to spell aix?? sco?? osf?? Nope, none of those.

The biggest blunder:

One morning I came in at my usual time of 6am, and had an operator ask me what was wrong with one of our production servers (servicing 6 banks). Well nothing worked at the console (it was already logged in as root). Even a "cat *" produced nothing but another shell prompt. I stopped and restarted the machine and when it attempted to come back up it didn't have any OS to run. Major issue, but we got our backup tapes from that night and restored the machine back to normal. I was clueless (sort of like today)

The next morning, the same operator caught me again, and this time I was getting angry (imagine that). Same crap, different day. Nothing was on any disk. This of course was before we had raid availble (not that that would have helped). So we restored the system from that nights backups and by 8am the banks have their systems up.

So now I have to fix this issue, but where the hell to start? I knew that production batch processing was done by 9PM, and that the backups started right after that. The backups completed around 1am, which were good backups, because we never lost a single transaction. But around 6am the stuff hit the fan. So I had a time frame: 1am-6am, something was clobbering the system. I went though the crons, but nothing really stood out, so I had to really dive into them. This is the code (well almost) I found in the script:

cd /tmp/uniplex/trash/garbage
rm -rf *

As soon as I saw those two lines, I realized that I was the one that had caused the system to crap out every morning. See, I needed some disk space, and I was doing some house cleaning, and I deleted the sub-directory "garbage" from the /tmp/uniplex/trash" directory. Of course the script is run by root, which attempted to "CD" to a non-existent directory, which failed, and cron was still cd'd to "/", it then proceeded to "rm -rf *" my system!

live free or die
harry

Bill Hassell

I guess my blunder sets the record for "most clobbered machines" in one day:

I created an inventory script to be used in the Response Center to track all the systems across the United States (about 320 systems). These are all test and problem replication machines but necessary for the R/C engineers to replicate customer problems.

The script was written about 1992 to handle version 7.0 and higher. About 1995, I had a number of useful scripts that it seemed reasonable to drop these into all 300 machines as a part of the inventory process (so far, so good). Then about that time, 10.01 was released and I made a few changes to the script. One was to change the useful script location from /usr/local/bin to /usr/contrib/bin because of bad directory permissions. I considered 'fixing' the bad permissions but since these systems must represent the customer environment, I decided to move everything.

Enter the shell option -u. I did not use that option in my scripts and due to a spelling error, an environment variable was used in rm -r which was null, thus removing the entire /usr/local directory on 320 machines overnight.

Needless to say, I never write scripts without set -u at the top of the script.

Dave Johnson

Here is my worst.
We us BC's on our XP512. We stop the application, resync the BC, split the BC, start the application, mount the BC on same server, start backup to tape from BC. Well I had to add a LUN to the primary and BC. I recreated the BC. I forgot to change the script that mounts the BC to include the new LUN. The error message vgimport when you do not include all the LUN's is just a warning and it makes the volume group available. The backups seemed to be working just fine.
Well 2 months go by. I did not have enough available disk space to test my backups. (That has been changed). Then I decided to be proactive about deleted old files. So I wrote a script:
cd /the/directory/I/want/to/thin/out
find . -mtime +30 -exec rm {} \;

Well that was scheduled on cron to run just before backups one night. The next morning I get the call the system is not responding. (I guessed later the cd command had failed and the find ran from /).
After a reboot I find lots of files are missing from /etc /var /usr /stand and so on. No problem, just rebuild from the make_recovery tape created 2 nights before then restore the rest from backup.
Well step 1 was fine, but the backup tape was bad. The database was incomplete. It took 3 days (that is 24 hours per day) to find the most recient tape with a valid database. Then we had to reload all the data. After the 3rd day I was able to turn over recovery to the developers. It took about a week to get the application back on-line.
I have sent a request to HP to have the vgimport command changed so a vgimport that does not specify all the LUN's will fail unless some new command line param is used. They have not yet provided this "enhancement" as of the last time I checked a couple of months ago. I now test for this condition and send mail to root as well as fail the BC mount if it does.

Life lesson: TEST YOUR BACKUPS!!

RAC

Well I was very very new to HP-UX. Wanted to set up PPP connection with a password borrowed from a friend so that I could browse the net.

Did not worry that the remote support modem can not dial out from remote support port.
Went through all documents available, created device files dozen times, but never worked. In anguish did

rm -fr `ltr|tail -4|Awk '{print $9}'

(That to pacify myself that I know complex commands)

But alas, I was /sbin/rc3.d.

Thought this is not going to work and left that.
Other colleage not aware of this rebooted the system for Veritas netbackup problem.

Within next two hours HP engineer was on-site. Was called by colleague.

Was watching whole recovery process, repeatedly saying "I want to learn, I want to learn"

Then came to know that can not be done.

RD

Ramkumar Devanathan

I deleted /etc/* once. (just a few config files, eh?? nope, had to reinstall the OS);-)

Dario

Got a page once at 4:00AM about a filesystem almost full. Got up and started working on it without checking if someone else was in the system. I was recalling my commands using ESC-k but my supervisor was removing files in a different directory using rm -r *. The rest is history.

Gary Cantwell

A trainee student of one of my customers deleted /sbin on a HP-UX 11 system and copied it back in from a HP-UX 10.20 system and tried to deny it...

He's no longer a trainee student :-)

Gary

YLTan

got another one;

Trying to be helpful....I use a find . -name *.log -exec rm -f {} \; to rid all unused log files in apps temp and log dir. but I got a bit creative and put this in root cronjobs and guess what!! the next day, some very angry DBA come banging...cos it removed all Oracle database re-do log files.

Spend the rest of the weekend restoring database....:(

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

bash - Safe Directory Delete - Unix & Linux Stack Exchange

Cleaning Up Your -tmp...The Safe Way Issue 18

Is it safe to delete stuff in /tmp (SUSE 10.2)

delete - Can I safely remove all the files in /tmp - Ask Ubuntu

linux - Delete all of -var-log - Server Fault



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: October 23, 2018