Creative uses of rm like rm -rf / tmp/* or rm -r /etc (instead of rm -r etc)

Unintended consequences, or wiping out much more then you intended to do as the  daily hazard of sysadmin  job

Version 2.1 (Nov 14, 2019)

Copyright 2006-2019, Nikolai Bezroukov

NOTE: This is copyrighted unpublished work based on original research by the author and presented as a preprint. All rights reserved.

• Introduction
• Probability that in the list of files to be deleted there are abnormal files dramatically increases with the number of file to be deleted
• Steps that you can do to reduce the chances of the major damage to the data
• Move files to be deleted to the trash bin as the first step and then delete files from trash several days later
• Use a wrapper like "saferm"
• Classic blunders with RM command
• Classic blunders in scripts with rm command
  • The deletion of wrong directory or set of directories due to uninitialized variable
  • Absence of checking of return code of commands that prepare rm environment (if such commands are used)
  • Not taking into consideration presence of files and/or directories with blanks in their names
  • Excessive optimisim; lack of testing
• Backup files before executing rm command if you deleting a directory and it is not that large
• More on blunders with the deleting a directory from the backup
• Important class of subtle Unix errors: Dot-star errors
• Using "convenience" symlinks to other (typically deeply nested) directories inside a directories and forgetting about them
• Disastrous typos in name of the directory or regex -- unintended space
• Root deletion protection
• Using echo command to see expansion of your wildcards, if any
• Using file starting with - as stoppers for unintended use of rm. Undeletable files
• Blunders in using rm with find

Introduction

The most enduring principles in the universe are accident and error

SNAFU is an acronym that is widely used to stand
for the sarcastic expression Situation Normal: All Fucked Up.
It is a well-known example of military acronym slang.
SNAFU - Wikipedia

FUBAR (Fucked/Fouled Up Beyond All Recognition/Any Repair/All Reason)
List of military slang terms

It is much better to read this page before you do something with rm then after you committed some inexcusable blunder :-)

Many Unix sysadmin horror stories are related to unintended consequences such as side effects of this classic Unix command. It is important to rm has several idiosyncrasies which are iether easily forgotten or unknown by novice sysadmins.  And the use of rm with find -exec option opens a new can of worms. Usually the disaster strikes due to some unfortunate and unique confluence of factors.  And, of course, 99.999% deletions using rm are successful and do not produce any side effects. This page is about remaining  0.001% or even 0.0001% which Anatoly Ivasyuk aptly named "Creative uses of rm" .  They while they are very rare, they are very destructive. Any of them, performed on a production box, can end sysadmin career in a particular organization. or at lest be a "career limiting event"

The main problem is this class of mishaps/SNAFU is that they happen very rarely and when people generally do not expect any troubles. Out of blue sky. Any seasoned sysadmins with decades of experience have their set of scars from blunders connected with misused  rm commands, but that does not mean that he periodically does not  repeat them. As such SNAFU happen very rarely (one in two-three years), the lessons learned are forgotten.  As Benjamin Franklin  quipped that "Experience keeps a dear school, but fools will learn in no other." But this saying is not applicable in this particular case: fool or no fool, if something bad happened three years ago you probably do not remember critical details now, or at the time you write your "one-time" scripts after running of which you discover some strange side effects ;-)

Such SNAFU belong to the  class of cases when  you simply can't learn from experience -- you need specialized and regular safety training.  Please consider rm as dangerous as a sharp blade, or a chainsaw.

You can commit rm blunders and experience corresponding SNAFU both directly on the command line and in your "one-time" scripts (usually shell or Perl scripts). The latter can be even more destructive then the command line blunders because on command line you usually realize that something is deeply wrong in a couple of seconds. If you run the script it can take much longer than that, or not detected until its execution finished. 

One aggravating factor is that rm -r is very fast on modern servers. It is I/O bound so the speed and number of cores of the CPU does not matter much, but presence of SSD greatly enhanced the damage. Even if the resulting list of files and directories is very long (which suspicious in itself and logically would need specifying an additional option like -I to execute) , rm starts working of it immediately one by one, left to right and  on a server with SSD disks you can easily wipe something between 200GB and one terabyte of data (depending on the average size of files)  just in several seconds which takes you to realize that the command that you submitted was very wrong.  For reference, around 10-20K files can be wiped in 5 sec -- the time usually needed to realize that submitted command is wrong and cancel it.

As rm actions change data stamp on directories affected by its actions. In case of deeply nested trees with large number of files in deeply nested directories you can see the damage in directories. Damaged directories  with wiped out subdirectories  now have current modification timestamp. Sometimes that helps to analyze the net result of this wild ride of rm against your data and acccess the scope of the damage. 

NOTE: it make sense to name directories with valuable data in large data trees Xcritical not Critical so that hopefully in case of SNAFY they were picked up later. 

In any case the only way to keep your "safety training" in rm blunders current is to use your own log and such pages as this one. they should be at the core of self study that you need to repeat at least once a year or before writing script with recognizes important data (for example converts folders in tar.gz files). 

It is much better to read this before you do something with rm then after you committed some inexcusable blunder :-)

One of the key problem here is too simplistic logic of rm command which definitely should  be re-designed in view of 50 year experience of using it in Unix. Some minor steps in this direction were made and  GNU rm does not allow to recursively deleted the root directory. But more can be done. That's why it makes sense to improve this  situation by using a wrapper, and alias it to rm command as those mishaps happened only in interactive sessions. I wrote my own Perl script saferm, while was initially based on safe-rm, but  then completely rewritten for the project were we need to deleted about 8TB of data located in a more then 1K of directories.   

Expansion of basic regular expression .* is one such case (its expansion includes .. which is a parent directory). This specifics of the behavior of rm -r .* can easily be forgotten from one encounter to another, especially if sysadmin works both with Unix and Windows servers. Consequences if executed as root are often tragic... And can cost a week of intensive work to fix.   

Generally, if you have enough space, it is better to move files first to the Trash folder then directly deleting them. 

In cases where there is not spare space to keep the files for a while using proper options also can help to limit the damage.  For example, as of April 2018Linux uses GNU implementation that has relatively new, and very useful option -I, which can replace older option -i as a better solution to the same problems. 

The utility rm  remains pretty Spartan, despite typical "option bloat" of many Linux utilities (see ls as an example). It has just a dozen of options, which each sysadmin should know by heart ;-). Here is the list:

-f, --force

ignore nonexistent files, never prompt

-i

prompt before every removal

-I

prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes

--interactive[=WHEN]

prompt according to WHEN: never, once (-I), or always (-i). Without WHEN, prompt always

--one-file-system

when removing a hierarchy recursively, skip any directory that is on a file system different from that of the corresponding command line argument

--no-preserve-root

do not treat '/' specially

--preserve-root

do not remove '/' (default)

-r, -R, --recursive

remove directories and their contents recursively

-v, --verbose

explain what is being done

--help

display this help and exit

--version

output version information and exit

Two of those options are unique to GNU version of rm and  are especially important:

• -I -- prompt once before removing more than three files, or when  removing recursively; less intrusive than -i, while still giving protection against most mistakes. Should be used instead if -i.

• --one-file-system -- do not traverse to other filesystems

NOTE: By default rm -r does not follow symlinks to directories. But if used in  find -exec rm {} \; does this as find traverse symlinked directories. that means that "convenience" symlinks at your home directory one day can play a very bad joke with you.

Probability that in the list of files to be deleted there are abnormal files dramatically increases with the number of file to be deleted

When you are trying to delete a large number of files you will face unique problems. In the number, say, is over 10K you will necessarily find files with completely unpredictable, or very creatively mangled names.  If the filesystem was damaged, and then (partially) recovered, some names in directories can have mind boggling combination of symbols.

Abnormal files are not only files with blanks in the name. There are many other anomalies that lead to unexpected results. And that is not limited to blanks in the file name -- classic anomaly when some files were transferred from Windows or created by Windows users. For example, names that  start with ~ (which will be interpreted as home directory corresponding to this particular name, unless in quotes), names that start with $ (in this case all the subsequent string of letters unless the filename is in single quote will be interpreted as a variable that most probably does not exists and that part of the name will be  reduced to zero string  with corresponding consequences.  Names that consist of dots only, or dots with some other special character.

Your jaw will drop when you see all the possibility to be creative in  assigning name to a file in Linux/Unix. Most of those anomalies typically are the result of some mistakes by the users, which  after their  creation did not bother to delete them (or have no knowledge to do this), but some may contain valuable information.

So in this case you need to create script that generate file with rm commands for each individual file. Even for several hundred thousand file this script on modern servers works reasonably fast, but it will fail on the line  in which it rm command encounts abnormal file for which you still did not make any precautions.  At least in this case you will get a diagnostic message with the line number.

In the script that generate rm commands to detect all the abnormal names and replaces them with their inode numbers. After that you need to filer all the command that looks like an attempt to delete directories, which is a typical side effect on abnormal files presence. You should never ever use option-r when deleting such a huge amount of files, or you will regret your decision. 

Writing special sanitizing script that generates one rm command for each file is the only way to go. There is no other way to deal with this situation without the risk of unexpected data destruction.

In sanitizing script you should check each name for the presence of abnormal symbols inside the file (which is a relatively easy task) as well as abnormal symbol combinations at the beginning of the file (file name starting with dot, ~, -, =, etc.  If you use Perl for preliminary checking of the list of files to be deleted, the detection of abnormal symbols can be easily accomplished using the tr command. Abnormal prefixes are more complex task. One way to deal with it is to list all file names that does not start with letter or digit,  extract all prefixes (say the first four letters of the name), sort them removing duplicates, and then investigate results.  That works even if you have millions of files to be deleted.

In any case, when you need to delete more then a 100K files your  jaw will drop when you see all the possibilities to be creative in  assigning name to a file

And please do not expect that you can accomplish such mass deletion fast (actually you can, but the result might be very unexpected ;-). This is a serious project with high risk of destruction of useful/valuable data and it needs to approached as such. It requires planning and investigation.

Steps that you can do to reduce the chances of the major damage to the data

 Above all, not too much zeal! (Talleyrand’s warning to young diplomats)

Talleyrand

Wiping out useful data by using rm with wrong parameters or options is probably the most common blunder committed both by novices and experienced sysadmins. The  only difference that experienced sysadmins usually have a backup.

In both case such a mishap happens very rarely, so unless you pay attention and organize for yourself some kind of monthly or quarterly "safety minutes" (which are typically a semi-useless monthly meetings typical for large corporate environment; they can be  made more useful for sysadmins, and IT staff in general,  if  they concentrate on sysadmin horror stores).

Without monthly "safety minutes" you usually forget about the dangers that  rm has and can accidentally step on the same rake again ;-).  As working with any sharp instrument, this danger can't be completely avoided, but in addition to having an up-to-date backup, there are several steps that you can do to reduce the chances of the major damage to the data:

1. If possible, do backup before any dangerous operation. Always do the backup of /etc on the server that you plan to work at the beginning of your working day. This can and probably should be done from dot scripts (such as .bash_profile)  on login.  Especially if you plan to delete files from such system directories such  as /etc,  /boot, /root, etc.  Or if the operation you intend to do is complex and requires traversing filesystem using of find -exec option or find -delete option. That's Rule No. 1
   • If you plan large data reorganization to safe space and do not have full backup you can always use multiple USB drives to create one. they are now are as large as 12 TB each.  Please remember that most painful data loses often happen during such reorganizations.  There is a saying that two data reorganizations have the same effect of one datacenter fire.
2. Use wrapper such as saferm of safe-rm.  It really helps to prevent  many typical cases of accidental deletion of important system and user files.
3. If you can't or do not want to use wrapper and plan to delete multiple files/directories using rm -r or  find -exec (see also Typical Errors In Using Find) please use non-destructive command like ls, tree, or find command   to check what files you are deleting first. Then visually verify correctness of your list.
   • In all more or less complex cases it make sense to delete files/directories as a separate step only after creating and  double checking of the list files to be deleted (sometimes you will be surprised what you will discover in the list created by supposedly "foolprof" regex ;-)  as the input to rm, for example

     ls -r my_super_clever_regex | tee /tmp/files_to_be_deleted
     more tee /tmp/files_to_be_deleted

     xargs -a /tmp/files_to_be_deleted

     or, better (if you have enough  space for files you plan to delete in /Trash or other filesystem)

        xargs -a /tmp/joeuser/my_delete_list -I % mv % /Trash  

   • Using find (with options -exec or -delete) for deletion introduces the whole new class of blunder that you need to be aware of. Please read corresponding page first Typical Errors In Using Find . See also Blunders in using rm with find below.

4. The same is true if you use regular expressions or traverse multiple filesystems. Please run ls, tree, or find command with the argument you intend to use in rm command  and  Wrappers for rm command do it automatically.
5. It is much safer to delete individual files and directories using file manager like WinSCP, or Midnight Commander, then do this directly by typing on the command line.  In this case you have a very important visual feedback, and this less likely to make a blunder. Also you do not need to type the name of file or directory to be deleted, which also help to avoid the whole class of errors.
6. When you are programming a simple script for file deletion, try to type in generated rm command (or any other destructive command) an absolute path to the directory on which you working with at least part of it specified expesidly like in rm /etc/myprogram/confi/*, not as rm $what_i_need_to_delete   In most cases writing commands like  rm $MYDIR is a very bad idea. Using some "safe" prefix is much better deal.  Something like rm /home/joeuser/$MYDIR This protects from cases when your variable for some reason is left undefined or you simply mistyped it pointing to the wrong directory.  One large class of rm blunders is when rm mistakenly operate on root,  or some system directory, or current directory.  Similarly, "chmod -R 755 /Backup/$mypath" is much safer then chmod -R 755 $mybackup
    
7. For small amount of files you can use move to Trash folder instead of deletion, if you have enough space for files that you are deleting if makes sense to delay actual wiping files out as long as possible. You can delete them from the Trash folder later on.
    
   • Many devastating blunders with rm occur when you think that you operate on backup copy of the  system, or important user/application directory, but in reality by mistake operate on a "real thing" or use system directory by  accidentally (and automatically) typing "/" in front of the name of common system directories such as /etc and  /var
   • Even worse the situation happens when you operate on wrong server.
      
   • Moving directories and files to /tmp/Trash folder is always a better deal if you have space.

     del () {
       now="$(date +%Y%m%d_%H%M%S)"
       mkdir -p /tmp/Trash/$now
       mv "$@" /tmp/Trash/$now"
     }

      For additional safety you can write your emptytrash function as:

     emptytrash() {
        rm -rf /tmp/Trash
        mkdir /tmp/Trash  
     }

     There is no way that function can delete anything, but the files you want it to.
      

   • If there are multiple directories with the same name or directories, which you plan to delete, have the same names as system directories (for example /Backup/etc) ,  you can first rename the directory. And only then apply rm -r command to the directory this this new, unique name, for example, /tmp/Junkdir (this way you are less likely mistype the name of the directory or accidentally put slash before its name as in rm /etc  instead of rm etc ...)
      
8. Use option -I instead of -i when deleting multiple files. First of all -i option (which it default alias in Red Hat and other distributions) is really annoying.  Moreover after a while the answer Yes became automatic, ingrained in your brain,  and that's a huge danger.  Many users start eventually simply disable it either by using option -f  or by specifying path, and thus avoiding using this alias. As option  -I is less intrusive it can be recommended as a better replacement. Unfortunately it does not solve that problem of reckless actions, but nothing probably can solve it. It prompts you only once before removing more than three files, or when removing recursively. See rm(1) remove files-directories:

   If the -I or --interactive=once option is given, and there are more than three files or the -r, -R, or --recursive are given, then rm prompts the user for whether to proceed with the entire operation. If the response is not affirmative, the entire command is aborted.

9. To delete subdirectories and files starting with dot you can use "shopt -s dotglob". 
10. Do not improvise complex, potentially destructive commands on the command line without necessity. History of command is your friend. If the command is complex or destructive it makes sense to write in the editor first, not directly improvise it on the command line.  Sometime while typing on command line  you can hit the Enter key or inserts a blank where you do not  intend. Using the editor help to avoid those blunders.
    • Do not improvise creating complex commands involving regular expressions on the command line. This is dangerous and not necessary
    • A weaker version of this rule is to write options and arguments of the command first and after verification add rm (or other dangerous utility name) in front.  
    • Sysadmins  tend automatically type the name of the most common systems directory with slash in front of them  instead of proper relative name (rm -r /etc instead of rm -r etc ) because they are conditioned to type it this way by a long practice and this sequence of keystroke is "etched" in your brain and you do it almost semi-consciously. If you first rename the directory to something else, for example DELETEIT first,  such an error will not occur . A script can check if there is another directory with this name in existence and warn you too.  You can also simply move directory to /Trash folder which has, say 90 day expiration period for files in it

NOTES:

• You also can make the whole /usr directory write-protected by using a separate filesystem for it (/usr/local should be symlinked to /opt where it belongs). There can be some other write protected filesystem as appropriate but usually this is "too little to late" kind of protection.
• You can also use extended attributes for extremely important/vital files.  For example, you can protect symlinks like /lib  and /lib64 in root directory on RHEL7 using expended attributes.
• If you operate server with SElinux in enforcing mode and know SElinux really well, you can use SElinux for file protection.

Move files to be deleted to the trash bin as the first step and then delete files from trash several days later.

One command line implementation is trash-cli. Another is rmtrash  Yet another is del

Autotrash helps to clean the trash in an easy way. None of those commands are installed by default. Ubuntu 18.04 can the command gio installed by default. Commanding gio help trash outputs:

Usage:
  gio trash [OPTION...] [LOCATION...]

Move files or directories to the trash.

Options:
  -f, --force     Ignore nonexistent files, never prompt
  --empty         Empty the trash

There is also libtrash which moves all deleted files automatically to the trash via LD_PRELOAD (but it seems to have several bugs).  The ideas used in libtrash are superiot to more simpistic alternatives. It's a pity it is buggy (and does not seem very active).

See shell - Make `rm` move to trash - Unix & Linux Stack Exchange for a useful discussion.

Use wrapper like saferm

There are two known to the author wrappers for rm:

• saferm -- written by the author and  uses "typed" regular expressions. Types can be l -link, f -- file , d -- directory and  a -- any (no type checking) . There is also type p similar  to a but in which the  comparison is performed using prefix string.  Written  in Perl. Uses tree utility as dependency.  It is oriented on sysadmins who usually know regular expression really well.
• safe-rm -- older and simpler wrapper also written in Perl which uses prefix strings.  Uses prefix strings for protection and as such can be used by users who do not know regular expressions. Also pretty effective

Both need to be  aliased to rm to be more effective. At least for the root.  Aliasing rm to the wrapper is safe because aliases work only in interactive sessions.

Wrapper saferm was written based on horror stories contained in this page and was inspired by safe-rm. It uses a set of Perl regular expression  to protect system directories and user files, which are more powerful and flexible then prefix strings used is safe-rm.  Both are better then nothing. Experience  has  shown that the set of regular expression that you can supply to it in configuration file is pretty powerful tool. It also makes some common sense checks like presence of ".." in the expansion of the arguments.

All key system directories should be protected from accidental deletion, as removal some of them make recovery process more complex. In RHEL7 that list includes also symlinks in the root directory like /lib64. And it goes without saying  that /etc or /usr should be protected better than other directories, with more protected regex defined. 

In addition, using saferm you can block the deletion of directories that are in your private "protected directories and files" list (by default stored in  ~/Saferm.saferm.conf ) .  There are also some indirect methods of checking if the directory you are trying to delete is a wrong directory. For example, if you use level 2 directory /Fav for links some often used directories, you can convert this list into regular expression and that guarantee that those directories will be protected from accidental deletion.

Additional random observations:

• Your home directory and /root often contains important files too and should not be neglected.
• If you use shell variables to specify the target of deletion, the key problem is that some of them can be undefined.  This is especially dangerous if you first cd to the target directory (regex - Safe rm -rf function in shell script - Stack Overflow )

  I've found a big danger with rm in bash is that bash usually doesn't stop for errors. That means that:

  cd $SOMEPATH
  rm -rf *

  Is a very dangerous combination if the change directory fails. A safer way would be:

  cd $SOMEPATH && rm -rf *

  EDIT: @placeybordeaux makes a good point that if $SOMEPATH is undefined or empty cd doesn't treat it as an error and returns 0. In light of that this answer should be considered unsafe unless $SOMEPATH is validated as existing and non-empty first. I believe cd with no args should be an illegal command since at best is performs a no-op and at worse it can lead to unexpected behaviour but it is what it is.

  Instead of cd $SOMEPATH, you should write cd "${SOMEPATH?}". The ${varname?} notation ensures that the expansion fails with a warning-message if the variable is unset or empty (such that the && ... part is never run); the double-quotes ensure that special characters in $SOMEPATH, such as whitespace, don't have undesired effects. – ruakh Jul 13 '18 at 6:46

  There is a set -u bash directive that will cause exit, when uninitialized variable is used. I read about it here, with rm -rf as an example. I think that's what you're looking for. And here is set's manual.

  I would recommend to use realpath(1) and not the command argument directly, so that you can avoid things like /A/B/../ or symbolic links.

Classic blunders with RM command

There are several classic blunders committed using the rm command :

1. Running rm - R command with regular expression, without testing regex expansion first by using ls . You should always use ls -Rl command to test complex rm -R  commands (  -R, --recursive means process  subdirectories recursively).
2. Mistyping complex path or file name in rm commands. It's always better to use copy and paste operation for directories and files used in rm command as it helps to avoid various typos. If you do recursive deletion is useful to have a "dry run" using find or ls to see the files.

   Here are to example of typos:

   rm -rf /tmp/foo/bar/ *

   instead of

   rm -rf /tmp/foo/bar/*

   ===

   Let's assume that you are in directory /backup that has  a subdirectory etc that you want to delete. As path "/etc" is ingrained in sysadmin mind it is very easy automatically/subconsciously type

   rm -r /etc

   instead of

   rm -r etc

   And realize  what you have done in a second or so. That's why for commands which include names OS system directories it is safer to type them in the editor, inspect them and only then execute that command on the command line.  If you are using terminal emulator on Windows desktop then Notepad++ or similar editor is OK.

   This is also why it is prudent to alias rm to the saferm script of your choice, which should prevent such mishaps (aliases are not used in non interactive sessions)

   This actually is an interesting type of "induced" error because /etc is typed daily so often that it kind of ingrained in sysadmin head and can be typed subconsciously instead of etc

3. Using option -r with  .*  regex without understanding its consequences.   That essentially make rm -r recursive and it traverses down to parent directories as this basic regular expression matches ".." (parent directory). 
4. You can accidentally hit Enter by mistake before finishing typing the line containing rm command.  If this is after the first character and the first character is /, or after several characters and the first are the  name of system directory like  /boot in older versions of Linux  you  would find yourself in trouble. Sometimes this happens with the new keyboard when you still do not fully adapted to new keys "feel".
5. You can accidentally insert a blank  after *, this separating the argument into two: "*" and the rest, with corresponding consequences.  Argument * probably should be allowed if and only if it is a single argument on the command  like unless -f is specified.  

Those cases can be viewed as shortcomings of rm implementation (For example, * should be allowed only as a single argument, not as one of several arguments, but Unix expands argument  before passing  them to command  (expansion is done  by shell and there is not simple access to the  original command line) so it is tricky to check; rm also should automatically block deletion of system directories like /,  /etc/ and the list of "protected" directories specified in its config file unless -f  flag is specified. Using has no system attribute so it is difficult to distinguish system files form the rest, but files owned by root:root probably deserve a special treatment as system files, even if one is working as root.

In other works implementation of rm does not take into account complexity of the current Unix environment and its power created an opportunity to wipe our more then you intended in way too many circumstances.

Also Unix does not have system attribute for files although sticky bit on files can be used instead along with ownership of sys instead of root).

Classic blunders in scripts with rm command

The deletion of wrong directory or set of directories due to uninitialized variable

The most classic blunder with rm is deletion of wrong directly due to uninitialized variable. For example

TARGET_DIR=$1
dir_name=$2
cd $TARGET_DIR; rm -r $TARGET_DIR/$DIR_NAME

As you can see the variable dir_name used in RM command was typed in all caps, but was defined and initialized in low case.

Here you need to use what is called Defensive programming. The main idea of which is to introduce additional "sanity" test before any important operation. One important check for script is bash setting -u which exits the script as soon as undefined variable is encountered.  And undefined variable are source of probably 50% SNAFU with rm in scripts.

For example

set -u 

if (( $# <2 )) ; then
    echo [ABEND] Less then two arguments were supplied
fi
TARGET_DIR=$1
sub_dir=$2
if [[ ! -d "$TARGET_DIR" ]] || [[ ! -d "$TARGET_DIR/$sub_dir" ]] ; then 
   echo [ABEND] $TARGET_DIR or $TARGET_DIR/$sub_dir are not valid subdirectories in /home
   exit 16
fi   
cd "/home/$TARGET_DIR" && rm -r "/home/$TARGET_DIR/$sub_dir"
if (( $? > 0 )); then 
   abend [ABEND] Abnormal return code from rm. Permissions problem ?
   exit 32
fi

Any script that use rm command but do not have a debug mode is an open  invitation to disaster. 

debug=1
if (( $debug > 0 )) ; then 
  SAFETY='I'
else 
  SAFETY=''
fi    

... ... ...

cd "/home/$TARGET_DIR" && rm -r$SAFETY "/home/$TARGET_DIR/$sub_dir"
if (( $? > 0 )); then 
   abend [ABEND] Abnormal return code from rm. Permissions problem ?
   exit 32
fi

 See also shell - How to check if a variable is set in Bash - Stack Overflow

Generalized approach to checking important parameters in subroutines and before passing them to external programs like rm is called defensive programming.unfortunatly Wikipedia article is extremely weak, but some ideas applicable for bash can be derived from articles devotes to defensive programming in Java. For example  Defensive programming the good, the bad and the ugly - Enterprise Craftsmanship

Checks above are example of assertions about the value of parameters. 

Absence of checking of return code of commands that prepare rm environment (if such commands are used)

The second common blunder in using rm command in scripts is absence of checking of return code of command that prepare rm environment (if such commands are used). For I While this is not recommended, is some cases cd command is used to make the  particular directory current before issuing the rm command. For example

cd /Etc/mc; rm *.conf

It is always safer to check the return code. For all Unix command return code 0 means success. So classic rewrite of the command above would be:

cd /Etc/mc && rm *.conf

cd Etc
if (( $? == 0 )) ; then 
   rm *.conf
   if (( $? > 0 )) ; then 
      echo [FAILURE] rm return code is $?
   fi
fi

cd Etc
if (( $? == 0 )) ; then 
   rm *.conf
   if (( $? > 0 )) ; then 
      echo [FAILURE] rm return code is $?
   fi
fi

Unfortunately not all GNU utilities document return code in man pages. You need to experiment  is such case to determines which situation result in which return code.

NOTE: The examples above are presented just for the illustration of the concept. Generally you should avoid executing rm command on the current directory in your scripts, because that increases possibility to execute the command in the current directory which is the one you intend to use.  Always provide and verify the directory as a parameter to rm

Not taking into consideration presence of files and/or directories with blanks in their names

If those files are transferred from windows and are never intended to be transferred back and replace original files in Windows it is prudent to replace all blanks with the underscore. Otherwise you will always need to walk on egg shells and enclose all variables that refer to directories and files in double quotes.  Using such files a bad idea everywhere, including Windows. And if you think that there can be no "creative" users who will name one of their files starting with space, think again ;-).  The situation with directories is much simpler: using directories with spaces in their names is actually a sign of criminal negligence, and should be prosecuted as such. Most users quickly understand the rules. But some do not.  For those recommendation to read The Bastard Operator From Hell  might  produce the necessary effect ;-)

There are multiple ways you can eliminate spaces in filenames and that procedure can be run periodically to ensure that the user comply with the rules with offending file communicated back to the users with appropriate subject line like "Gross violation  of the naming conversions of the server XXX" ;-) as otherwise those changes created fake "missing file" tickets.

They were discussed in-depth in the following Stack Overflow article: linux - How to replace spaces in file names using a bash script - Stack Overflow as well as the post by Steven Vona (May 27, 2019) The Rename Command - Renaming Files In Linux. Here is one of the solution that uses only classic Unix utilities:

armandino, Dec 3, 2013 at 20:51

find . -depth -name '* *' | while IFS= read -r f ; do mv -i "$f" "$(dirname "$f")/$(basename "$f"|tr ' ' _)" ; done

failed to get it right at first, because I didn't think of directories.

Note usage of -depth directive to manage traversal performed by find. It's necessary. I just want to point out that in Linux  there are multiple options

• detox(1) clean up filenames - Linux man page The detox utility renames files to make them easier to work with. It removes spaces and other such annoyances. It'll also translate or cleanup Latin-1 (ISO 8859-1) characters encoded in 8-bit ASCII, Unicode characters encoded in UTF-8, and CGI escaped characters.  See also https://github.com/dharple/detox
• rename  another binary utility which allow to change one string in the file name into (often installed on Linux system by default; (it has  a small man page)
  • Dennis Williamson ,Mar 22, 2012 at 20:33

    A find/rename solution. rename is part of util-linux.

    You need to descend depth first, because a whitespace filename can be part of a whitespace directory:

    find /tmp/ -depth -name "* *" -execdir rename " " "_" "{}" ";"

     

• A Perl script called prename which allow regular expressions to be used for renaming. See `prename` script from Debian's 'perl' package · GitHub. prename (original name: rename) is a perl script written by Larry Wall to rename files using Perl code. It's been around at least since the Perl 4 days and lived in eg/rename in the perl source code, but has since split in the Perl 5 source code. See example of usage at perl - Rename multiple files from command line - Stack Overflow

Jürgen Hötzel ,Nov 4, 2015 at 3:03

Use rename (aka prename ) which is a Perl script which may be on your system already. Do it in two steps:

find -name "* *" -type d | prename 's/ /_/g'    # do the directories first
find -name "* *" -type f | prename 's/ /_/g'

Based on Jürgen's answer and able to handle multiple layers of files and directories in a single bound using the "Revision 1.5 1998/12/18 16:16:31 rmb1" version of /usr/bin/rename (a Perl script):

find /tmp/ -depth  -name "* *" -execdir prename 's/ /_/g' "{}" \;

Excessive optimisim; lack of testing

If anything can go wrong, it will.
Murphy Law

In case the script uses rm -r as a part of its operation the lack of testing is completely inexcusable and is a sign of lack of professionalism. Even very short scripts can contain grave errors which in retrospect would be revealed even by a simple test.

Such scripts should also have the debug variable defined and non zero value of this this variable  should provide some additional debugging information in testing mode. On high values of debug execution of the commands can be replaced with echoing their content. See Defensive programming

Is you issue all the command via special subroutine (let's call it run) this mode is very easy to implement.

Also log in such cases should be written to a file as sometime usage of screen mangle the STDOUT/STDERR output.  The easiet way to do it is to create an envelope in which yu can use iether  exec command or tee command. For example:

if [[ -d /tmp/myscript ]] ; then
   mkdir /tmp/myscript
fi
my_script /etc/mc 2>&1 | tee /tmp/myscript.log

Backup files before executing any "large scale" rm command, especially if you deleting files in system directory or an important files in use of data directories and the total size is not that large

"Make back of /etc first"  is the Rule No.1 if you delete something  from /etc directory or other system directories (/boot, /root, etc). Generally you should create the tarball of /etc directory at the beginning of the first root session for the particular day. Making tarball of the /etc directory on modern servers takes a couple of seconds.

Backups of subtrees affected before execution of rm command are also important. can save quite a lot of nerves in situations that otherwise can s can be devastating. For example, in the example above you would erase all your files and subdirectories in the /etc directory. Modern flavors of Unix usually prevent erasing /,  but not /etc. I think Red hat version  of GNU rm prevents erasing all level 2 system directories, but I am not sure.

Any attempt to remove a file whose last file name component is . or .. is rejected without any prompting, as mandated by POSIX.

If we are talking about more then 1GB of files, you can also move directories to /Trash folder and delete them after, say 7 days or so.  Later you can delete files that are say, older then 7 days using a cron script, or you can delay deletion until the total size of this directory exceeds a certain threshold. 

Within 10TB limit, You can always buy USB 3.0 drive and use it for such operations. Even if you buy it on your own money it is much cheaper than a couple of sleepless nights trying to recover lost information. 

There no excuse not to provide a backup in case of any large scale deletion.

More on blunders with the deleting a directory from the backup

Experience keeps a dear school, but fools will learn in no other. ~Benjamin Franklin

The saying is that experience keeps the most expensive school but fools are unable to learn in any other ;-).

It is dangerous to type destructive command directly on the command line as there are some idiosyncrasies that can play with you a very bad joke. Often backup has directories with the same name as systems directories simply because this is a backup of system directories. I once automatically typed /etc instead of etc trying to delete directory /backup/etc or something like that  to free space on a backup directory on a production server (/etc probably in engraved in sysadmin head as it is typed so often and can be substituted for etc subconsciously).  I realized that it was mistake and cancelled the command, but it was a fast server and one third of /etc was gone. The rest of the day was spoiled...  Actually not completely: I learned quite a bit about the behavior of AIX in this situation and the structure of AIX /etc directory this day, so each such disaster is actually a great learning experience, almost like one day training course ;-).

But it's much less nerve wracking to get this knowledge from the course... Another interesting thing is having backup was not enough is this case -- enterprise backup software stopped working on a damaged server. The same was true for telnet and ssh. And this was a remote server in a datacenter across the country.  I restored the directory on the other non-production server (first overwriting its /etc directory in this second box with the help of operations, tell me about cascading errors and Murphy law :-). In this case one ssh  session survives and helped to transfer the tar file.  As netcat does not requires authentication and if it is statically linked  it might help in such cases as you might be able to transfer missing files using it.

Such blunders are really persistent as often used directories are types in a "semiconscious" fashion, in "autopilot" mode and you realize the blunder only after you hit Enter. 

In such cases network services with authentication stop working the only way to transfer files is using CD/DVD, USB drive (if cp works, which in RHEL is not given as it is dynamically linked) or netcat. That's why it is useful to have netcat on servers: netcat is the last resort file transfer program when  services with authentication like ftp or scp stop working.  It is especially useful to have it if the datacenter is remote.

A simple extra space often produced horrible consequences:

cd /usr/lib
ls /tmp/foo/bar

I typed

rm -rf /tmp/foo/bar/ *

instead of

rm -rf /tmp/foo/bar/*

The system doesn’t run very will without all of it’s libraries……

NOTE: You can block such behavior requiring that if -r option was given you should have one and only one argument to rm. That probably should be a part of functionality of your "saferm" script.

Important class of subtle Unix errors: dot-star errors

A popular class of recursive rm errors are so called dot-star-errors. They happen when rm is using with -r option or when rm is used  in find -exec option.  Novice sysadmins usually do not realize that '.*' also matches '..' often with disastrous consequences.  If you are in any doubt as for how a wildcard will expand use echo command to see the result.

• From: [email protected] (Contractor Bob Johnson)
  Organization: Tinker Air Force Base, Oklahoma

  Another horror story (mine this time)...

  Cleaning out an old directory, I did 'rm *', then noticed several files that began with dot (.profile, etc) still there.  So, in a fit of obtuse brilliance, I typed...

      rm -rf .* &

  By the time I got it stopped, it had chewed through 3 filesystems which all had to be restored from tape (.* expands to ../*, and the -r makes it keep walking up the directory tree).  Live and learn...
   

• My mistake on SunOS (with OpenWindows) was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed

  "rm -rf ./.*".

  I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first.

Using "convenience" symlinks to other (typically deeply nested) directories inside a directories and forgetting about them

Command  rm does not follows symlinks but find does.  So if you have a symlink to some system on important directory from the directory you are deleting using

find -exec rm {} \;

you are up to nasty surprise

That's the main danger of "convenience symlinks" which are used to simplify access to deeply nested directories.  They are more dangerous then useful. Using aliases or functions is a better deal.

If you use them, do this only from the level two directory created specially for this purpose like /Fav and protect this directory in your saferm script.

Disastrous typos in name of the directory or regex -- unintended space

There are some exotic typos that can lean you to troubles, especially with -r  option. One of them is unintended space:

 rm /home/joeuser/old *

instead of

rm /home/joeuser/old*

Dumb expression... Gradual realization of what you have done. Cold sweat..

Wrapper can catch some of those errors if  /home/joeuser/old does not exists.

Or, similarly:

 rm * _old

instead of

rm *_old

Such a mistake is even more damaging  is you use -r option. For example:

rm -rf /tmp/foo/bar/ *

instead of

rm -rf /tmp/foo/bar/*

So it is prudent to block execution of rm commands with multiple argument your saferm script, if option  -r is specified.

Root deletion protection

To remove a file you must have write permission on the folder where it is stored. Sun introduced "rm -rf /" protection in Solaris 10, first released in 2005. Upon executing the command, the system now reports that the removal of / is not allowed.

Shortly after, the same functionality was introduced into FreeBSD version of rm  utility.

GNU rm  refuses to execute rm -rf /  if the --preserve-root  option is given, which has been the default since version 6.4 of GNU Core Utilities was released in 2006.

No such protection exists for critical system directories like /etc /bin, and so on but you can imitate it putting file "-i" into such directories or using a wrapper for the interactive usage of the command. 

Using echo command to see expansion of your wildcards, if any

To understand what will be actually being executed after shell expansion, preface your rm command with echo. For example, if there is a filename starting with the dash you will receive very confusing message from rm

	$ rm *
	rm: unknown option -- -
	usage: rm [-f|-i] [-dPRrvW] file ...

That side effect can be used for protection of system directories. To find out what caused it prefix the command with echo

echo rm *

You can generate list of files dynamically for rm (piping it via xargs if it is too large.

But a better way is to include  generation and analysis of the list of files be  deleted into your saferm script and switch to generation of the list instead of actual deletion is any "suspicious files or circumstances are detected.

alias rm="saferm"

Please note that alias used in RHEL (alias rm="rm -i"), unfortunately, encourages a tendency to mindlessly pound y  and the return key to affirm removes - until you realize that got just past the file you need to keep.

Using a file starting with - as stoppers for unintended use of rm. Undeletable files

The rm command accepts the -- option which will cause it to stop processing flag options from that point forward. This allows the removal of file names that begin with a dash (-).

rm -- -filename

Actually accidental creation of files starting with dash is a common problem for novice sysadmin.  The problem is that if I type rm -foo, the rm command treats the filename as an option.

There are two simple ways to delete such a file. One was shown above and based on supplying an empty option argument. The second to use a relative or absolute pathname including at least one directory in  front:

rm ./--protected

rm /home/user/--protected

Here the debugging trick that we mentioned above -- prefacing rm command with echo really helps to understand what happens.

Deletion of files with non-printable characters often lead to unintended consequences

The safest way to do this is to use inodes for such files. You can also specify regular expression replacing non-printable characters in the name with ?  for each bad character. For example:

rm bad?file?name

But trying " a shortcut" which does not require meticulously putting ? in the name like" 

rm bad* 

Another way to delete files that have control characters in the name is to  use  -i option and an asterisk, which gives you the option of removing each file in the directory — even the ones that you can't type.

% rm -i *
rm: remove faq.html (y/n)? n
rm: remove foo (y/n)? y
%

The -i option may also be helpful when you are dealing with files with Unicode characters that appear to be regular letters in your locale, but that don't match patterns or names you use otherwise.

You can also create the function for deletion of files with special characters. Something along the lines:

# cat  /root/del_special_characters.sh
   if [ ! -d /tmp/Special_files ] ; then
      mkdir /tmp/Special_files
   fi
   genfile=/tmp/Special_files/list`date +"%y%m%d"`
   echo dev/null > $genfile
   find . -maxdepth 1  -type f -ls | fgrep 'a' | while  read myline
   do
      echo Processing: $myline
      IFS=' ' # Warning: the IFS variable means split by one of characters in a set
      set $myline
      inode=$1
      echo inode=$inode
      new_name=${!#};
      new_name=`tr 'a' '_' <<< "$new_name"`
      echo "new_name=$new_name"
      echo mv $inode $new_name
      echo mv $inode $new_name >> $genfile
   done
   echo "Generated commands:"
   cat $genfile

A great way to discover files with control characters in them is to use the -q option to the Unix ls command (some systems also support a useful -b option). You can, for example, alias the ls command:

alias lsq='ls -liq '

Files that have control characters in their filenames will then appear with question marks:

lqs f*
faq.html                fo?o

Blunders in using rm with find

This is class or blunders that deserves a page of its own. See Typical Errors In Using Find  for more information about this SNAFU

You need to be extra careful running rm in an -exec option of find command. There are many horror stories in which people delete large parts of their filesystem due to typos of sloppy find command parameters. Again it is prudent first to test it using ls command to see the actual list of files to be deleted. Sometimes it is quite different from what you intended to do :-).

you can also generate the list using ls command, inspect it in a separate step. And only then pipe it into rm :

cat rm.lst | xargs -t -n1 rm 

Dr. Nikolai Bezroukov

NEWS CONTENTS

• 20191108 : How to prevent and recover from accidental file deletion in Linux Enable Sysadmin ( Nov 08, 2019 , www.redhat.com )
• 20191108 : My first sysadmin mistake by Jim Hall ( Nov 08, 2019 , opensource.com )
• 20191108 : How to use Sanoid to recover from data disasters Opensource.com ( Nov 08, 2019 , opensource.com )
• 20191107 : How to prevent and recover from accidental file deletion in Linux Enable Sysadmin ( Nov 07, 2019 , www.redhat.com )
• 20191025 : Get inode number of a file on linux - Fibrevillage ( Oct 25, 2019 , www.fibrevillage.com )
• 20191025 : Howto Delete files by inode number by Erik ( Feb 10, 2011 , erikimh.com )
• 20191025 : unix - Remove a file on Linux using the inode number - Super User ( Oct 25, 2019 , superuser.com )
• 20191025 : Linux - Unix Find Inode Of a File Command ( Jun 21, 2012 , www.cyberciti.biz )
• 20190904 : Basic Trap for File Cleanup ( Sep 04, 2019 , www.putorius.net )
• 20190826 : linux - Avoiding accidental 'rm' disasters - Super User ( Aug 26, 2019 , superuser.com )
• 20190826 : bash - How to prevent rm from reporting that a file was not found ( Aug 26, 2019 , stackoverflow.com )
• 20190826 : shell - rm -rf return codes ( Aug 26, 2019 , superuser.com )
• 20190221 : https://github.com/MikeDacre/careful_rm ( Feb 21, 2019 , github.com )
• 20190221 : https://github.com/lagerspetz/linux-stuff/blob/master/scripts/saferm.sh by Eemil Lagerspetz ( Feb 21, 2019 , github.com )
• 20190221 : The rm='rm -i' alias is an horror ( Feb 21, 2019 , superuser.com )
• 20190221 : Ubuntu Manpage trash - Command line trash utility. ( Feb 21, 2019 , manpages.ubuntu.com )
• 20190128 : "Right," I said. "Time to get the backup." I knew I had to leave when I saw his face start twitching and he whispered: "Backup ...?" ( Jan 28, 2019 , opensource.com )
• 20190128 : regex - Safe rm -rf function in shell script ( Jan 28, 2019 , stackoverflow.com )
• 20181022 : linux - If I rm -rf a symlink will the data the link points to get erased, to ( Oct 22, 2018 , unix.stackexchange.com )
• 20181022 : Does rm -rf follow symbolic links? ( Jan 25, 2012 , superuser.com )
• 20181005 : Sometimes one extra space makes a big difference ( Oct 05, 2018 , cam.ac.uk )
• 20181005 : Deletion of files purpose of which you do not understand sometimes backfire by Anatoly Ivasyuk ( Oct 05, 2018 , cam.ac.uk )
• 20181005 : Danger of hidden symlinks ( Oct 05, 2018 , cam.ac.uk )
• 20181005 : Hidden symlinks and recursive deletion of the directories ( Oct 05, 2018 , www.reddit.com )
• 20181005 : Automatically putting slash in front of directory named like system (named like bin,etc,usr, var) which are all etched in sysadmin memory ( Oct 05, 2018 , www.reddit.com )
• 20181002 : Rookie almost wipes customer's entire inventory unbeknownst to sysadmin ( Oct 02, 2018 , theregister.co.uk )
• 20180422 : THE classic Unix horror story programming ( Apr 22, 2008 , www.reddit.com )
• 20180422 : rm and Its Dangers (Unix Power Tools, 3rd Edition) ( Apr 22, 2018 , docstore.mik.ua )
• 20180422 : How to prevent a mistaken rm -rf for specific folders? ( Jan 20, 2013 , unix.stackexchange.com )
• 20180422 : bash - Safe Directory Delete ( Mar 13, 2013 , Stack Exchange )
• 20180422 : bash - Is it safe to alias rm to safe-rm ( Jun 7, 2014 , Ask Ubuntu )
• 20180422 : THE classic Unix horror story WSU Linux Users Group ( THE classic Unix horror story WSU Linux Users Group, )
• 20180422 : Unix Recovery Legend ( )
• 20180422 : This classic article from Mario Wolczko first appeared on Usenet in 1986. ( )
• 20180421 : Any alias of rm is a very stupid idea ( Feb 14, 2017 , www.cyberciti.biz )
• 20170720 : The ULTIMATE Horrors story with recovery! ( Nov 08, 2002 , www.linuxjournal.com )
• 20100612 : Sysadmin Blunder (3rd Response) ( Toolbox for IT Groups )
• 20100612 : Creative uses of rm ( Creative uses of rm, )
• 20100612 : Theres an interesting thread on the CentOS mailing list on sys admin horror stories. Amusing read. ( )
• 20100612 : [Ilugc] [OT] sys admin horror stories ( [Ilugc] [OT] sys admin horror stories, )
• 20100612 : G ( IT Resource Center )

Old News ;-)

[Nov 08, 2019] How to prevent and recover from accidental file deletion in Linux Enable Sysadmin

trashy - Trashy · GitLab might make sense in simple cases. But often massive file deletions are about attempts to get free space.

Nov 08, 2019 | www.redhat.com

Back up

You knew this would come first. Data recovery is a time-intensive process and rarely produces 100% correct results. If you don't have a backup plan in place, start one now.

Better yet, implement two. First, provide users with local backups with a tool like rsnapshot . This utility creates snapshots of each user's data in a ~/.snapshots directory, making it trivial for them to recover their own data quickly.

There are a great many other open source backup applications that permit your users to manage their own backup schedules.

Second, while these local backups are convenient, also set up a remote backup plan for your organization. Tools like AMANDA or BackupPC are solid choices for this task. You can run them as a daemon so that backups happen automatically.

Backup planning and preparation pay for themselves in both time, and peace of mind. There's nothing like not needing emergency response procedures in the first place.

Ban rm

On modern operating systems, there is a Trash or Bin folder where users drag the files they don't want out of sight without deleting them just yet. Traditionally, the Linux terminal has no such holding area, so many terminal power users have the bad habit of permanently deleting data they believe they no longer need. Since there is no "undelete" command, this habit can be quite problematic should a power user (or administrator) accidentally delete a directory full of important data.

Many users say they favor the absolute deletion of files, claiming that they prefer their computers to do exactly what they tell them to do. Few of those users, though, forego their rm command for the more complete shred , which really removes their data. In other words, most terminal users invoke the rm command because it removes data, but take comfort in knowing that file recovery tools exist as a hacker's un- rm . Still, using those tools take up their administrator's precious time. Don't let your users -- or yourself -- fall prey to this breach of logic.

If you really want to remove data, then rm is not sufficient. Use the shred -u command instead, which overwrites, and then thoroughly deletes the specified data

However, if you don't want to actually remove data, don't use rm . This command is not feature-complete, in that it has no undo feature, but has the capacity to be undone. Instead, use trashy or trash-cli to "delete" files into a trash bin while using your terminal, like so:

$ trash ~/example.txt
$ trash --list
example.txt

One advantage of these commands is that the trash bin they use is the same your desktop's trash bin. With them, you can recover your trashed files by opening either your desktop Trash folder, or through the terminal.

If you've already developed a bad rm habit and find the trash command difficult to remember, create an alias for yourself:

$ echo "alias rm='trash'"

Even better, create this alias for everyone. Your time as a system administrator is too valuable to spend hours struggling with file recovery tools just because someone mis-typed an rm command.

Respond efficiently

Unfortunately, it can't be helped. At some point, you'll have to recover lost files, or worse. Let's take a look at emergency response best practices to make the job easier. Before you even start, understanding what caused the data to be lost in the first place can save you a lot of time:

• If someone was careless with their trash bin habits or messed up dangerous remove or shred commands, then you need to recover a deleted file.
• If someone accidentally overwrote a partition table, then the files aren't really lost. The drive layout is.
• In the case of a dying hard drive, recovering data is secondary to the race against decay to recover the bits themselves (you can worry about carving those bits into intelligible files later).

No matter how the problem began, start your rescue mission with a few best practices:

• Stop using the drive that contains the lost data, no matter what the reason. The more you do on this drive, the more you risk overwriting the data you're trying to rescue. Halt and power down the victim computer, and then either reboot using a thumb drive, or extract the damaged hard drive and attach it to your rescue machine.
• Do not use the victim hard drive as the recovery location. Place rescued data on a spare volume that you're sure is working. Don't copy it back to the victim drive until it's been confirmed that the data has been sufficiently recovered.
• If you think the drive is dying, your first priority after powering it down is to obtain a duplicate image, using a tool like ddrescue or Clonezilla .

Once you have a sense of what went wrong, It's time to choose the right tool to fix the problem. Two such tools are Scalpel and TestDisk , both of which operate just as well on a disk image as on a physical drive.

Practice (or, go break stuff)

At some point in your career, you'll have to recover data. The smart practices discussed above can minimize how often this happens, but there's no avoiding this problem. Don't wait until disaster strikes to get familiar with data recovery tools. After you set up your local and remote backups, implement command-line trash bins, and limit the rm command, it's time to practice your data recovery techniques.

Download and practice using Scalpel, TestDisk, or whatever other tools you feel might be useful. Be sure to practice data recovery safely, though. Find an old computer, install Linux onto it, and then generate, destroy, and recover. If nothing else, doing so teaches you to respect data structures, filesystems, and a good backup plan. And when the time comes and you have to put those skills to real use, you'll appreciate knowing what to do.

[Nov 08, 2019] My first sysadmin mistake by Jim Hall

Wiping out of /etc directory is one thing that sysadmin accidentally do. This is often happen if the other directory is name etc, for example /Backup/etc. In such cases you automatically put a slash in front of etc because it is ingrained in your mind. And you put the slash in front of etc subconsciously, not realizing what you are doing. And then faces consequences. If you do not use saferm, the result are pretty devastating. In most cases the sever does not die, but new logins are impossible. SSH session survives. That's why it is important to backup /etc/at the first login to the server. On modern severs it takes a couple of seconds.

If subdirectories are intact then you still can copy the content from another server. But content of sysconfig subdirectory in linux is unique to the server and you need a backup to restore it.

Notable quotes:

"... As root. I thought I was deleting some stale cache files for one of our programs. Instead, I wiped out all files in the /etc directory by mistake. Ouch. ..."

"... I put together a simple strategy: Don't reboot the server. Use an identical system as a template, and re-create the ..."

Nov 08, 2019 | opensource.com

rm command in the wrong directory. As root. I thought I was deleting some stale cache files for one of our programs. Instead, I wiped out all files in the /etc directory by mistake. Ouch.

My clue that I'd done something wrong was an error message that rm couldn't delete certain subdirectories. But the cache directory should contain only files! I immediately stopped the rm command and looked at what I'd done. And then I panicked. All at once, a million thoughts ran through my head. Did I just destroy an important server? What was going to happen to the system? Would I get fired?

Fortunately, I'd run rm * and not rm -rf * so I'd deleted only files. The subdirectories were still there. But that didn't make me feel any better.

Immediately, I went to my supervisor and told her what I'd done. She saw that I felt really dumb about my mistake, but I owned it. Despite the urgency, she took a few minutes to do some coaching with me. "You're not the first person to do this," she said. "What would someone else do in your situation?" That helped me calm down and focus. I started to think less about the stupid thing I had just done, and more about what I was going to do next.

I put together a simple strategy: Don't reboot the server. Use an identical system as a template, and re-create the /etc directory.

Once I had my plan of action, the rest was easy. It was just a matter of running the right commands to copy the /etc files from another server and edit the configuration so it matched the system. Thanks to my practice of documenting everything, I used my existing documentation to make any final adjustments. I avoided having to completely restore the server, which would have meant a huge disruption.

To be sure, I learned from that mistake. For the rest of my years as a systems administrator, I always confirmed what directory I was in before running any command.

I also learned the value of building a "mistake strategy." When things go wrong, it's natural to panic and think about all the bad things that might happen next. That's human nature. But creating a "mistake strategy" helps me stop worrying about what just went wrong and focus on making things better. I may still think about it, but knowing my next steps allows me to "get over it."

[Nov 08, 2019] How to use Sanoid to recover from data disasters Opensource.com

Nov 08, 2019 | opensource.com

filesystem-level snapshot replication to move data from one machine to another, fast . For enormous blobs like virtual machine images, we're talking several orders of magnitude faster than rsync .

If that isn't cool enough already, you don't even necessarily need to restore from backup if you lost the production hardware; you can just boot up the VM directly on the local hotspare hardware, or the remote disaster recovery hardware, as appropriate. So even in case of catastrophic hardware failure , you're still looking at that 59m RPO, <1m RTO.

https://www.youtube.com/embed/5hEixXutaPo

Backups -- and recoveries -- don't get much easier than this.

The syntax is dead simple:

root@box1:~# syncoid pool/images/vmname root@box2:pooln
ame/images/vmname

Or if you have lots of VMs, like I usually do... recursion!

root@box1:~# syncoid -r pool/images/vmname root@box2:po
olname/images/vmname

This makes it not only possible, but easy to replicate multiple-terabyte VM images hourly over a local network, and daily over a VPN. We're not talking enterprise 100mbps symmetrical fiber, either. Most of my clients have 5mbps or less available for upload, which doesn't keep them from automated, nightly over-the-air backups, usually to a machine sitting quietly in an owner's house.

Preventing your own Humpty Level Events

Sanoid is open source software, and so are all its dependencies. You can run Sanoid and Syncoid themselves on pretty much anything with ZFS. I developed it and use it on Linux myself, but people are using it (and I support it) on OpenIndiana, FreeBSD, and FreeNAS too.

You can find the GPLv3 licensed code on the website (which actually just redirects to Sanoid's GitHub project page), and there's also a Chef Cookbook and an Arch AUR repo available from third parties.

[Nov 07, 2019] How to prevent and recover from accidental file deletion in Linux Enable Sysadmin

trashy - Trashy · GitLab might make sense in simple case. But often deletions are about increasing free space.

Nov 07, 2019 | www.redhat.com

Back up

You knew this would come first. Data recovery is a time-intensive process and rarely produces 100% correct results. If you don't have a backup plan in place, start one now.

Better yet, implement two. First, provide users with local backups with a tool like rsnapshot . This utility creates snapshots of each user's data in a ~/.snapshots directory, making it trivial for them to recover their own data quickly.

There are a great many other open source backup applications that permit your users to manage their own backup schedules.

Second, while these local backups are convenient, also set up a remote backup plan for your organization. Tools like AMANDA or BackupPC are solid choices for this task. You can run them as a daemon so that backups happen automatically.

Backup planning and preparation pay for themselves in both time, and peace of mind. There's nothing like not needing emergency response procedures in the first place.

Ban rm

On modern operating systems, there is a Trash or Bin folder where users drag the files they don't want out of sight without deleting them just yet. Traditionally, the Linux terminal has no such holding area, so many terminal power users have the bad habit of permanently deleting data they believe they no longer need. Since there is no "undelete" command, this habit can be quite problematic should a power user (or administrator) accidentally delete a directory full of important data.

Many users say they favor the absolute deletion of files, claiming that they prefer their computers to do exactly what they tell them to do. Few of those users, though, forego their rm command for the more complete shred , which really removes their data. In other words, most terminal users invoke the rm command because it removes data, but take comfort in knowing that file recovery tools exist as a hacker's un- rm . Still, using those tools take up their administrator's precious time. Don't let your users -- or yourself -- fall prey to this breach of logic.

If you really want to remove data, then rm is not sufficient. Use the shred -u command instead, which overwrites, and then thoroughly deletes the specified data

However, if you don't want to actually remove data, don't use rm . This command is not feature-complete, in that it has no undo feature, but has the capacity to be undone. Instead, use trashy or trash-cli to "delete" files into a trash bin while using your terminal, like so:

$ trash ~/example.txt
$ trash --list
example.txt

One advantage of these commands is that the trash bin they use is the same your desktop's trash bin. With them, you can recover your trashed files by opening either your desktop Trash folder, or through the terminal.

If you've already developed a bad rm habit and find the trash command difficult to remember, create an alias for yourself:

$ echo "alias rm='trash'"

Even better, create this alias for everyone. Your time as a system administrator is too valuable to spend hours struggling with file recovery tools just because someone mis-typed an rm command.

Respond efficiently

Unfortunately, it can't be helped. At some point, you'll have to recover lost files, or worse. Let's take a look at emergency response best practices to make the job easier. Before you even start, understanding what caused the data to be lost in the first place can save you a lot of time:

• If someone was careless with their trash bin habits or messed up dangerous remove or shred commands, then you need to recover a deleted file.
• If someone accidentally overwrote a partition table, then the files aren't really lost. The drive layout is.
• In the case of a dying hard drive, recovering data is secondary to the race against decay to recover the bits themselves (you can worry about carving those bits into intelligible files later).

No matter how the problem began, start your rescue mission with a few best practices:

• Stop using the drive that contains the lost data, no matter what the reason. The more you do on this drive, the more you risk overwriting the data you're trying to rescue. Halt and power down the victim computer, and then either reboot using a thumb drive, or extract the damaged hard drive and attach it to your rescue machine.
• Do not use the victim hard drive as the recovery location. Place rescued data on a spare volume that you're sure is working. Don't copy it back to the victim drive until it's been confirmed that the data has been sufficiently recovered.
• If you think the drive is dying, your first priority after powering it down is to obtain a duplicate image, using a tool like ddrescue or Clonezilla .

Once you have a sense of what went wrong, It's time to choose the right tool to fix the problem. Two such tools are Scalpel and TestDisk , both of which operate just as well on a disk image as on a physical drive.

Practice (or, go break stuff)

At some point in your career, you'll have to recover data. The smart practices discussed above can minimize how often this happens, but there's no avoiding this problem. Don't wait until disaster strikes to get familiar with data recovery tools. After you set up your local and remote backups, implement command-line trash bins, and limit the rm command, it's time to practice your data recovery techniques.

Download and practice using Scalpel, TestDisk, or whatever other tools you feel might be useful. Be sure to practice data recovery safely, though. Find an old computer, install Linux onto it, and then generate, destroy, and recover. If nothing else, doing so teaches you to respect data structures, filesystems, and a good backup plan. And when the time comes and you have to put those skills to real use, you'll appreciate knowing what to do.

[Oct 25, 2019] Get inode number of a file on linux - Fibrevillage

Oct 25, 2019 | www.fibrevillage.com

Get inode number of a file on linux

An inode is a data structure in UNIX operating systems that contains important information pertaining to files within a file system. When a file system is created in UNIX, a set amount of inodes is created, as well. Usually, about 1 percent of the total file system disk space is allocated to the inode table.

How do we find a file's inode ?

ls -i Command: display inode

ls -i Command: display inode
$ls -i /etc/bashrc
131094 /etc/bashrc
131094 is the inode of /etc/bashrc.

Stat Command: display Inode

$stat /etc/bashrc
  File: `/etc/bashrc'
  Size: 1386          Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 131094      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-12-10 10:01:29.509908811 -0800
Modify: 2013-06-06 11:31:51.792356252 -0700
Change: 2013-06-06 11:31:51.792356252 -0700

find command: display inode

$find ./ -iname sysfs_fc_tools.tar -printf '%p %i\n'
./sysfs_fc_tools.tar 28311964

Notes :

    %p stands for file path
    %i stands for inode number

tree command: display inode under a directory

#tree -a -L 1 --inodes /etc
/etc
├── [ 132896]  a2ps
├── [ 132898]  a2ps.cfg
├── [ 132897]  a2ps-site.cfg
├── [ 133315]  acpi
├── [ 131864]  adjtime
├── [ 132340]  akonadi
...

usecase of using inode

find / -inum XXXXXX -print to find the full path for each file pointing to inode XXXXXX.

Though you can use the example to do rm action, but simply I discourage to do so, for security concern in find command, also in other file system, same inode refers a very different file.

filesystem repair

If you get a bad luck on your filesystem, most of time, run fsck to fix it. It helps if you have inode info of the filesystem in hand.
This is another big topic, I'll have another article for it.

[Oct 25, 2019] Howto Delete files by inode number by Erik

Feb 10, 2011 | erikimh.com

linux administration - tips, notes and projects

6 Comments

Ever mistakenly pipe output to a file with special characters that you couldn't remove?

-rw-r–r– 1 eriks eriks 4 2011-02-10 22:37 –fooface

Good luck. Anytime you pass any sort of command to this file, it's going to interpret it as a flag. You can't fool rm, echo, sed, or anything else into actually deeming this a file at this point. You do, however, have a inode for every file.

Traditional methods fail:

[eriks@jaded: ~]$ rm -f –fooface
rm: unrecognized option '–fooface'
Try `rm ./–fooface' to remove the file `–fooface'.
Try `rm –help' for more information.
[eriks@jaded: ~]$ rm -f '–fooface'
rm: unrecognized option '–fooface'
Try `rm ./–fooface' to remove the file `–fooface'.
Try `rm –help' for more information.

So now what, do you live forever with this annoyance of a file sitting inside your filesystem, never to be removed or touched again? Nah.

We can remove a file, simply by an inode number, but first we must find out the file inode number:

$ ls -il | grep foo

Output:

[eriks@jaded: ~]$ ls -il | grep foo
508160 drwxr-xr-x 3 eriks eriks 4096 2010-10-27 18:13 foo3
500724 -rw-r–r– 1 eriks eriks 4 2011-02-10 22:37 –fooface
589907 drwxr-xr-x 2 eriks eriks 4096 2010-11-22 18:52 tempfoo
589905 drwxr-xr-x 2 eriks eriks 4096 2010-11-22 18:48 tmpfoo

The number you see prior to the file permission set is actually the inode # of the file itself.

Hint: 500724 is inode number we want removed.

Now use find command to delete file by inode:

# find . -inum 500724 -exec rm -i {} \;

There she is.

[eriks@jaded: ~]$ find . -inum 500724 -exec rm -i {} \;
rm: remove regular file `./–fooface'? y

[Oct 25, 2019] unix - Remove a file on Linux using the inode number - Super User

Oct 25, 2019 | superuser.com

,

ome other methods include:

escaping the special chars:

[~]$rm \"la\*

use the find command and only search the current directory. The find command can search for inode numbers, and has a handy -delete switch:

[~]$ls -i
7404301 "la*

[~]$find . -maxdepth 1 -type f -inum 7404301
./"la*

[~]$find . -maxdepth 1 -type f -inum 7404301 -delete
[~]$ls -i
[~]$

,

Maybe I'm missing something, but...

rm '"la*'

Anyways, filenames don't have inodes, files do. Trying to remove a file without removing all filenames that point to it will damage your filesystem.

[Oct 25, 2019] Linux - Unix Find Inode Of a File Command

Jun 21, 2012 | www.cyberciti.biz

... ... ..

stat Command: Display Inode

You can also use the stat command as follows:
$ stat fileName-Here
$ stat /etc/passwd
Sample outputs:

  File: `/etc/passwd'
  Size: 1644            Blocks: 8          IO Block: 4096   regular file
Device: fe01h/65025d    Inode: 25766495    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-05-05 16:29:42.000000000 +0530
Modify: 2012-05-05 16:29:20.000000000 +0530
Change: 2012-05-05 16:29:21.000000000 +0530

Share on Facebook Twitter

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter .

[Sep 04, 2019] Basic Trap for File Cleanup

Sep 04, 2019 | www.putorius.net

Basic Trap for File Cleanup

Using an trap to cleanup is simple enough. Here is an example of using trap to clean up a temporary file on exit of the script.

#!/bin/bash
trap "rm -f /tmp/output.txt" EXIT
yum -y update > /tmp/output.txt
if grep -qi "kernel" /tmp/output.txt; then
     mail -s "KERNEL UPDATED" [email protected] < /tmp/output.txt
fi

NOTE: It is important that the trap statement be placed at the beginning of the script to function properly. Any commands above the trap can exit and not be caught in the trap.

Now if the script exits for any reason, it will still run the rm command to delete the file. Here is an example of me sending SIGINT (CTRL+C) while the script was running.

# ./test.sh
 ^Cremoved '/tmp/output.txt'

NOTE: I added verbose ( -v ) output to the rm command so it prints "removed". The ^C signifies where I hit CTRL+C to send SIGINT.

This is a much cleaner and safer way to ensure the cleanup occurs when the script exists. Using EXIT ( 0 ) instead of a single defined signal (i.e. SIGINT – 2) ensures the cleanup happens on any exit, even successful completion of the script.

[Aug 26, 2019] linux - Avoiding accidental 'rm' disasters - Super User

Aug 26, 2019 | superuser.com

Avoiding accidental 'rm' disasters Ask Question Asked 6 years, 3 months ago Active 6 years, 3 months ago Viewed 1k times 1

Mr_Spock ,May 26, 2013 at 11:30

Today, using sudo -s , I wanted to rm -R ./lib/ , but I actually rm -R /lib/ .

I had to reinstall my OS (Mint 15) and re-download and re-configure all my packages. Not fun.

How can I avoid similar mistakes in the future?

Vittorio Romeo ,May 26, 2013 at 11:55

First of all, stop executing everything as root . You never really need to do this. Only run individual commands with sudo if you need to. If a normal command doesn't work without sudo, just call sudo !! to execute it again.

If you're paranoid about rm , mv and other operations while running as root, you can add the following aliases to your shell's configuration file:

[ $UID = 0 ] && \
  alias rm='rm -i' && \
  alias mv='mv -i' && \
  alias cp='cp -i'

These will all prompt you for confirmation ( -i ) before removing a file or overwriting an existing file, respectively, but only if you're root (the user with ID 0).

Don't get too used to that though. If you ever find yourself working on a system that doesn't prompt you for everything, you might end up deleting stuff without noticing it. The best way to avoid mistakes is to never run as root and think about what exactly you're doing when you use sudo .

[Aug 26, 2019] bash - How to prevent rm from reporting that a file was not found

Aug 26, 2019 | stackoverflow.com

How to prevent rm from reporting that a file was not found? Ask Question Asked 7 years, 4 months ago Active 1 year, 4 months ago Viewed 101k times 133 19

---

pizza ,Apr 20, 2012 at 21:29

I am using rm within a BASH script to delete many files. Sometimes the files are not present, so it reports many errors. I do not need this message. I have searched the man page for a command to make rm quiet, but the only option I found is -f , which from the description, "ignore nonexistent files, never prompt", seems to be the right choice, but the name does not seem to fit, so I am concerned it might have unintended consequences.

• Is the -f option the correct way to silence rm ? Why isn't it called -q ?
• Does this option do anything else?

Keith Thompson ,Dec 19, 2018 at 13:05

The main use of -f is to force the removal of files that would not be removed using rm by itself (as a special case, it "removes" non-existent files, thus suppressing the error message).

You can also just redirect the error message using

$ rm file.txt 2> /dev/null

(or your operating system's equivalent). You can check the value of $? immediately after calling rm to see if a file was actually removed or not.

vimdude ,May 28, 2014 at 18:10

Yes, -f is the most suitable option for this.

tripleee ,Jan 11 at 4:50

-f is the correct flag, but for the test operator, not rm

[ -f "$THEFILE" ] && rm "$THEFILE"

this ensures that the file exists and is a regular file (not a directory, device node etc...)

mahemoff ,Jan 11 at 4:41

\rm -f file will never report not found.

Idelic ,Apr 20, 2012 at 16:51

As far as rm -f doing "anything else", it does force ( -f is shorthand for --force ) silent removal in situations where rm would otherwise ask you for confirmation. For example, when trying to remove a file not writable by you from a directory that is writable by you.

Keith Thompson ,May 28, 2014 at 18:09

I had same issue for cshell. The only solution I had was to create a dummy file that matched pattern before "rm" in my script.

[Aug 26, 2019] shell - rm -rf return codes

Aug 26, 2019 | superuser.com

rm -rf return codes Ask Question Asked 6 years ago Active 6 years ago Viewed 15k times 8 0

---

SheetJS ,Aug 15, 2013 at 2:50

Any one can let me know the possible return codes for the command rm -rf other than zero i.e, possible return codes for failure cases. I want to know more detailed reason for the failure of the command unlike just the command is failed(return other than 0).

Adrian Frühwirth ,Aug 14, 2013 at 7:00

To see the return code, you can use echo $? in bash.

To see the actual meaning, some platforms (like Debian Linux) have the perror binary available, which can be used as follows:

$ rm -rf something/; perror $?
rm: cannot remove `something/': Permission denied
OS error code   1:  Operation not permitted

rm -rf automatically suppresses most errors. The most likely error you will see is 1 (Operation not permitted), which will happen if you don't have permissions to remove the file. -f intentionally suppresses most errors

Adrian Frühwirth ,Aug 14, 2013 at 7:21

grabbed coreutils from git....

looking at exit we see...

openfly@linux-host:~/coreutils/src $ cat rm.c | grep -i exit
  if (status != EXIT_SUCCESS)
  exit (status);
  /* Since this program exits immediately after calling 'rm', rm need not
  atexit (close_stdin);
          usage (EXIT_FAILURE);
        exit (EXIT_SUCCESS);
          usage (EXIT_FAILURE);
        error (EXIT_FAILURE, errno, _("failed to get attributes of %s"),
        exit (EXIT_SUCCESS);
  exit (status == RM_ERROR ? EXIT_FAILURE : EXIT_SUCCESS);

Now looking at the status variable....

openfly@linux-host:~/coreutils/src $ cat rm.c | grep -i status
usage (int status)
  if (status != EXIT_SUCCESS)
  exit (status);
  enum RM_status status = rm (file, &x);
  assert (VALID_STATUS (status));
  exit (status == RM_ERROR ? EXIT_FAILURE : EXIT_SUCCESS);

looks like there isn't much going on there with the exit status.

I see EXIT_FAILURE and EXIT_SUCCESS and not anything else.

so basically 0 and 1 / -1

To see specific exit() syscalls and how they occur in a process flow try this

openfly@linux-host:~/ $ strace rm -rf $whatever

fairly simple.

ref:

http://www.unix.com/man-page/Linux/EXIT_FAILURE/exit/

[Feb 21, 2019] https://github.com/MikeDacre/careful_rm

Feb 21, 2019 | github.com

rm is a powerful *nix tool that simply drops a file from the drive index. It doesn't delete it or put it in a Trash can, it just de-indexes it which makes the file hard to recover unless you want to put in the work, and pretty easy to recover if you are willing to spend a few hours trying (use shred to actually secure erase files).

careful_rm.py is inspired by the -I interactive mode of rm and by safe-rm . safe-rm adds a recycle bin mode to rm, and the -I interactive mode adds a prompt if you delete more than a handful of files or recursively delete a directory. ZSH also has an option to warn you if you recursively rm a directory.

These are all great, but I found them unsatisfying. What I want is for rm to be quick and not bother me for single file deletions (so rm -i is out), but to let me know when I am deleting a lot of files, and to actually print a list of files that are about to be deleted . I also want it to have the option to trash/recycle my files instead of just straight deleting them.... like safe-rm , but not so intrusive (safe-rm defaults to recycle, and doesn't warn).

careful_rm.py is fundamentally a simple rm wrapper, that accepts all of the same commands as rm , but with a few additional options features. In the source code CUTOFF is set to 3 , so deleting more files than that will prompt the user. Also, deleting a directory will prompt the user separately with a count of all files and subdirectories within the folders to be deleted.

Furthermore, careful_rm.py implements a fully integrated trash mode that can be toggled on with -c . It can also be forced on by adding a file at ~/.rm_recycle , or toggled on only for $HOME (the best idea), by ~/.rm_recycle_home . The mode can be disabled on the fly by passing --direct , which forces off recycle mode.

The recycle mode tries to find the best location to recycle to on MacOS or Linux, on MacOS it also tries to use Apple Script to trash files, which means the original location is preserved (note Applescript can be slow, you can disable it by adding a ~/.no_apple_rm file, but Put Back won't work). The best location for trashes goes in this order:

1. $HOME/.Trash on Mac or $HOME/.local/share/Trash on Linux
2. <mountpoint>/.Trashes on Mac or <mountpoint>/.Trash-$UID on Linux
3. /tmp/$USER_trash

Always the best trash can to avoid Volume hopping is favored, as moving across file systems is slow. If the trash does not exist, the user is prompted to create it, they then also have the option to fall back to the root trash ( /tmp/$USER_trash ) or just rm the files.

/tmp/$USER_trash is almost always used for deleting system/root files, but note that you most likely do not want to save those files, and straight rm is generally better.

[Feb 21, 2019] https://github.com/lagerspetz/linux-stuff/blob/master/scripts/saferm.sh by Eemil Lagerspetz

Shell script that tires to implement trash can idea

Feb 21, 2019 | github.com

#!/bin/bash

##

## saferm.sh

## Safely remove files, moving them to GNOME/KDE trash instead of deleting.

## Made by Eemil Lagerspetz

## Login <vermind@drache>

##

## Started on Mon Aug 11 22:00:58 2008 Eemil Lagerspetz

## Last update Sat Aug 16 23:49:18 2008 Eemil Lagerspetz

##

version= " 1.16 " ;

... ... ...

[Feb 21, 2019] The rm='rm -i' alias is an horror

Feb 21, 2019 | superuser.com

The rm='rm -i' alias is an horror because after a while using it, you will expect rm to prompt you by default before removing files. Of course, one day you'll run it with an account that hasn't that alias set and before you understand what's going on, it is too late.

... ... ...

If you want save aliases, but don't want to risk getting used to the commands working differently on your system than on others, you can to disable rm like this

alias rm='echo "rm is disabled, use remove or trash or /bin/rm instead."'

Then you can create your own safe alias, e.g.

alias remove='/bin/rm -irv'

or use trash instead.

[Feb 21, 2019] Ubuntu Manpage trash - Command line trash utility.

Feb 21, 2019 | manpages.ubuntu.com

xenial ( 1 ) trash.1.gz

Provided by: trash-cli_0.12.9.14-2_all

NAME

       trash - Command line trash utility.
SYNOPSIS 

       trash [arguments] ...
DESCRIPTION 

       Trash-cli  package  provides  a command line interface trashcan utility compliant with the
       FreeDesktop.org Trash Specification.  It remembers the name, original path, deletion date,
       and permissions of each trashed file.

ARGUMENTS 

       Names of files or directory to move in the trashcan.

EXAMPLES

       $ cd /home/andrea/
       $ touch foo bar
       $ trash foo bar
BUGS 

       Report bugs to http://code.google.com/p/trash-cli/issues

AUTHORS

       Trash  was  written  by Andrea Francia <[email protected]> and Einar Orn
       Olason <[email protected]>.  This manual page was written by  Steve  Stalcup  <[email protected]>.
       Changes made by Massimo Cavalleri <[email protected]>.

SEE ALSO 

       trash-list(1),   trash-restore(1),   trash-empty(1),   and   the   FreeDesktop.org   Trash
       Specification at http://www.ramendik.ru/docs/trashspec.html.

       Both are released under the GNU General Public License, version 2 or later.

[Jan 28, 2019] "Right," I said. "Time to get the backup." I knew I had to leave when I saw his face start twitching and he whispered: "Backup ...?"

Jan 28, 2019 | opensource.com

SemperOSS on 13 Sep 2016 Permalink This one seems to be a classic too:

Working for a large UK-based international IT company, I had a call from newest guy in the internal IT department: "The main server, you know ..."

"Yes?"

"I was cleaning out somebody's homedir ..."

"Yes?"

"Well, the server stopped running properly ..."

"Yes?"

"... and I can't seem to get it to boot now ..."

"Oh-kayyyy. I'll just totter down to you and give it an eye."

I went down to the basement where the IT department was located and had a look at his terminal screen on his workstation. Going back through the terminal history, just before a hefty amount of error messages, I found his last command: 'rm -rf /home/johndoe /*'. And I probably do not have to say that he was root at the time (it was them there days before sudo, not that that would have helped in his situation).

"Right," I said. "Time to get the backup." I knew I had to leave when I saw his face start twitching and he whispered: "Backup ...?"

==========

Bonus entries from same company:

It was the days of the 5.25" floppy disks (Wikipedia is your friend, if you belong to the younger generation). I sometimes had to ask people to send a copy of a floppy to check why things weren't working properly. Once I got a nice photocopy and another time, the disk came with a polite note attached ... stapled through the disk, to be more precise!

[Jan 28, 2019] regex - Safe rm -rf function in shell script

Jan 28, 2019 | stackoverflow.com

---

community wiki
5 revs
,May 23, 2017 at 12:26

This question is similar to What is the safest way to empty a directory in *nix?

I'm writing bash script which defines several path constants and will use them for file and directory manipulation (copying, renaming and deleting). Often it will be necessary to do something like:

rm -rf "/${PATH1}"
rm -rf "${PATH2}/"*

While developing this script I'd want to protect myself from mistyping names like PATH1 and PATH2 and avoid situations where they are expanded to empty string, thus resulting in wiping whole disk. I decided to create special wrapper:

rmrf() {
    if [[ $1 =~ "regex" ]]; then
        echo "Ignoring possibly unsafe path ${1}"
        exit 1
    fi

    shopt -s dotglob
    rm -rf -- $1
    shopt -u dotglob
}

Which will be called as:

rmrf "/${PATH1}"
rmrf "${PATH2}/"*

Regex (or sed expression) should catch paths like "*", "/*", "/**/", "///*" etc. but allow paths like "dir", "/dir", "/dir1/dir2/", "/dir1/dir2/*". Also I don't know how to enable shell globbing in case like "/dir with space/*". Any ideas?

EDIT: this is what I came up with so far:

rmrf() {
    local RES
    local RMPATH="${1}"
    SAFE=$(echo "${RMPATH}" | sed -r 's:^((\.?\*+/+)+.*|(/+\.?\*+)+.*|[\.\*/]+|.*/\.\*+)$::g')
    if [ -z "${SAFE}" ]; then
        echo "ERROR! Unsafe deletion of ${RMPATH}"
        return 1
    fi

    shopt -s dotglob
    if [ '*' == "${RMPATH: -1}" ]; then
        echo rm -rf -- "${RMPATH/%\*/}"*
        RES=$?
    else
        echo rm -rf -- "${RMPATH}"
        RES=$?
    fi
    shopt -u dotglob

    return $RES
}

Intended use is (note an asterisk inside quotes):

rmrf "${SOMEPATH}"
rmrf "${SOMEPATH}/*"

where $SOMEPATH is not system or /home directory (in my case all such operations are performed on filesystem mounted under /scratch directory).

CAVEATS:

• not tested very well
• not intended to use with paths possibly containing '..' or '.'
• should not be used with user-supplied paths
• rm -rf with asterisk probably can fail if there are too many files or directories inside $SOMEPATH (because of limited command line length) - this can be fixed with 'for' loop or 'find' command

SpliFF ,Jun 14, 2009 at 13:45

I've found a big danger with rm in bash is that bash usually doesn't stop for errors. That means that:

cd $SOMEPATH
rm -rf *

Is a very dangerous combination if the change directory fails. A safer way would be:

cd $SOMEPATH && rm -rf *

Which will ensure the rf won't run unless you are really in $SOMEPATH. This doesn't protect you from a bad $SOMEPATH but it can be combined with the advice given by others to help make your script safer.

EDIT: @placeybordeaux makes a good point that if $SOMEPATH is undefined or empty cd doesn't treat it as an error and returns 0. In light of that this answer should be considered unsafe unless $SOMEPATH is validated as existing and non-empty first. I believe cd with no args should be an illegal command since at best is performs a no-op and at worse it can lead to unexpected behaviour but it is what it is.

Sazzad Hissain Khan ,Jul 6, 2017 at 11:45

nice trick, I am one stupid victim. – Sazzad Hissain Khan Jul 6 '17 at 11:45

placeybordeaux ,Jun 21, 2018 at 22:59

If $SOMEPATH is empty won't this rm -rf the user's home directory? – placeybordeaux Jun 21 '18 at 22:59

SpliFF ,Jun 27, 2018 at 4:10

@placeybordeaux The && only runs the second command if the first succeeds - so if cd fails rm never runs – SpliFF Jun 27 '18 at 4:10

placeybordeaux ,Jul 3, 2018 at 18:46

@SpliFF at least in ZSH the return value of cd $NONEXISTANTVAR is 0 – placeybordeaux Jul 3 '18 at 18:46

ruakh ,Jul 13, 2018 at 6:46

Instead of cd $SOMEPATH , you should write cd "${SOMEPATH?}" . The ${varname?} notation ensures that the expansion fails with a warning-message if the variable is unset or empty (such that the && ... part is never run); the double-quotes ensure that special characters in $SOMEPATH , such as whitespace, don't have undesired effects. – ruakh Jul 13 '18 at 6:46

community wiki
2 revs
,Jul 24, 2009 at 22:36

There is a set -u bash directive that will cause exit, when uninitialized variable is used. I read about it here , with rm -rf as an example. I think that's what you're looking for. And here is set's manual .

,Jun 14, 2009 at 12:38

I think "rm" command has a parameter to avoid the deleting of "/". Check it out.

Max ,Jun 14, 2009 at 12:56

Thanks! I didn't know about such option. Actually it is named --preserve-root and is not mentioned in the manpage. – Max Jun 14 '09 at 12:56

Max ,Jun 14, 2009 at 13:18

On my system this option is on by default, but it cat't help in case like rm -ri /* – Max Jun 14 '09 at 13:18

ynimous ,Jun 14, 2009 at 12:42

I would recomend to use realpath(1) and not the command argument directly, so that you can avoid things like /A/B/../ or symbolic links.

Max ,Jun 14, 2009 at 13:30

Useful but non-standard command. I've found possible bash replacement: archlinux.org/pipermail/pacman-dev/2009-February/008130.html – Max Jun 14 '09 at 13:30

Jonathan Leffler ,Jun 14, 2009 at 12:47

Generally, when I'm developing a command with operations such as ' rm -fr ' in it, I will neutralize the remove during development. One way of doing that is:

RMRF="echo rm -rf"
...
$RMRF "/${PATH1}"

This shows me what should be deleted - but does not delete it. I will do a manual clean up while things are under development - it is a small price to pay for not running the risk of screwing up everything.

The notation ' "/${PATH1}" ' is a little unusual; normally, you would ensure that PATH1 simply contains an absolute pathname.

Using the metacharacter with ' "${PATH2}/"* ' is unwise and unnecessary. The only difference between using that and using just ' "${PATH2}" ' is that if the directory specified by PATH2 contains any files or directories with names starting with dot, then those files or directories will not be removed. Such a design is unlikely and is rather fragile. It would be much simpler just to pass PATH2 and let the recursive remove do its job. Adding the trailing slash is not necessarily a bad idea; the system would have to ensure that $PATH2 contains a directory name, not just a file name, but the extra protection is rather minimal.

Using globbing with ' rm -fr ' is usually a bad idea. You want to be precise and restrictive and limiting in what it does - to prevent accidents. Of course, you'd never run the command (shell script you are developing) as root while it is under development - that would be suicidal. Or, if root privileges are absolutely necessary, you neutralize the remove operation until you are confident it is bullet-proof.

Max ,Jun 14, 2009 at 13:09

To delete subdirectories and files starting with dot I use "shopt -s dotglob". Using rm -rf "${PATH2}" is not appropriate because in my case PATH2 can be only removed by superuser and this results in error status for "rm" command (and I verify it to track other errors). – Max Jun 14 '09 at 13:09

Jonathan Leffler ,Jun 14, 2009 at 13:37

Then, with due respect, you should use a private sub-directory under $PATH2 that you can remove. Avoid glob expansion with commands like 'rm -rf' like you would avoid the plague (or should that be A/H1N1?). – Jonathan Leffler Jun 14 '09 at 13:37

Max ,Jun 14, 2009 at 14:10

Meanwhile I've found this perl project: http://code.google.com/p/safe-rm/

community wiki
too much php ,Jun 15, 2009 at 1:55

If it is possible, you should try and put everything into a folder with a hard-coded name which is unlikely to be found anywhere else on the filesystem, such as ' foofolder '. Then you can write your rmrf() function as:

rmrf() {
    rm -rf "foofolder/$PATH1"
    # or
    rm -rf "$PATH1/foofolder"
}

There is no way that function can delete anything but the files you want it to.

vadipp ,Jan 13, 2017 at 11:37

Actually there is a way: if PATH1 is something like ../../someotherdir – vadipp Jan 13 '17 at 11:37

community wiki
btop ,Jun 15, 2009 at 6:34

You may use

set -f    # cf. help set

to disable filename generation (*).

community wiki
Howard Hong ,Oct 28, 2009 at 19:56

You don't need to use regular expressions.
Just assign the directories you want to protect to a variable and then iterate over the variable. eg:

protected_dirs="/ /bin /usr/bin /home $HOME"
for d in $protected_dirs; do
    if [ "$1" = "$d" ]; then
        rm=0
        break;
    fi
done
if [ ${rm:-1} -eq 1 ]; then
    rm -rf $1
fi

,

Add the following codes to your ~/.bashrc

# safe delete
move_to_trash () { now="$(date +%Y%m%d_%H%M%S)"; mv "$@" ~/.local/share/Trash/files/"$@_$now"; }
alias del='move_to_trash'

# safe rm
alias rmi='rm -i'

Every time you need to rm something, first consider del , you can change the trash folder. If you do need to rm something, you could go to the trash folder and use rmi .

One small bug for del is that when del a folder, for example, my_folder , it should be del my_folder but not del my_folder/ since in order for possible later restore, I attach the time information in the end ( "$@_$now" ). For files, it works fine.

[Oct 22, 2018] linux - If I rm -rf a symlink will the data the link points to get erased, to

Notable quotes:

"... Put it in another words, those symlink-files will be deleted. The files they "point"/"link" to will not be touch. ..."

Oct 22, 2018 | unix.stackexchange.com

user4951 ,Jan 25, 2013 at 2:40

This is the contents of the /home3 directory on my system:

./   backup/    hearsttr@  lost+found/  randomvi@  sexsmovi@
../  freemark@  investgr@  nudenude@    romanced@  wallpape@

I want to clean this up but I am worried because of the symlinks, which point to another drive.

If I say rm -rf /home3 will it delete the other drive?

John Sui

rm -rf /home3 will delete all files and directory within home3 and home3 itself, which include symlink files, but will not "follow"(de-reference) those symlink.

Put it in another words, those symlink-files will be deleted. The files they "point"/"link" to will not be touch.

[Oct 22, 2018] Does rm -rf follow symbolic links?

Jan 25, 2012 | superuser.com

I have a directory like this:

$ ls -l
total 899166
drwxr-xr-x 12 me scicomp       324 Jan 24 13:47 data
-rw-r--r--  1 me scicomp     84188 Jan 24 13:47 lod-thin-1.000000-0.010000-0.030000.rda
drwxr-xr-x  2 me scicomp       808 Jan 24 13:47 log
lrwxrwxrwx  1 me scicomp        17 Jan 25 09:41 msg -> /home/me/msg

And I want to remove it using rm -r .

However I'm scared rm -r will follow the symlink and delete everything in that directory (which is very bad).

I can't find anything about this in the man pages. What would be the exact behavior of running rm -rf from a directory above this one?

LordDoskias Jan 25 '12 at 16:43, Jan 25, 2012 at 16:43

How hard it is to create a dummy dir with a symlink pointing to a dummy file and execute the scenario? Then you will know for sure how it works! –

hakre ,Feb 4, 2015 at 13:09

X-Ref: If I rm -rf a symlink will the data the link points to get erased, too? ; Deleting a folder that contains symlinks – hakre Feb 4 '15 at 13:09

Susam Pal ,Jan 25, 2012 at 16:47

Example 1: Deleting a directory containing a soft link to another directory.

susam@nifty:~/so$ mkdir foo bar
susam@nifty:~/so$ touch bar/a.txt
susam@nifty:~/so$ ln -s /home/susam/so/bar/ foo/baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── foo
    └── baz -> /home/susam/so/bar/

3 directories, 1 file
susam@nifty:~/so$ rm -r foo
susam@nifty:~/so$ tree
.
└── bar
    └── a.txt

1 directory, 1 file
susam@nifty:~/so$

So, we see that the target of the soft-link survives.

Example 2: Deleting a soft link to a directory

susam@nifty:~/so$ ln -s /home/susam/so/bar baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── baz -> /home/susam/so/bar

2 directories, 1 file
susam@nifty:~/so$ rm -r baz
susam@nifty:~/so$ tree
.
└── bar
    └── a.txt

1 directory, 1 file
susam@nifty:~/so$

Only, the soft link is deleted. The target of the soft-link survives.

Example 3: Attempting to delete the target of a soft-link

susam@nifty:~/so$ ln -s /home/susam/so/bar baz
susam@nifty:~/so$ tree
.
├── bar
│   └── a.txt
└── baz -> /home/susam/so/bar

2 directories, 1 file
susam@nifty:~/so$ rm -r baz/
rm: cannot remove 'baz/': Not a directory
susam@nifty:~/so$ tree
.
├── bar
└── baz -> /home/susam/so/bar

2 directories, 0 files

The file in the target of the symbolic link does not survive.

The above experiments were done on a Debian GNU/Linux 9.0 (stretch) system.

Wyrmwood ,Oct 30, 2014 at 20:36

rm -rf baz/* will remove the contents – Wyrmwood Oct 30 '14 at 20:36

Buttle Butkus ,Jan 12, 2016 at 0:35

Yes, if you do rm -rf [symlink], then the contents of the original directory will be obliterated! Be very careful. – Buttle Butkus Jan 12 '16 at 0:35

frnknstn ,Sep 11, 2017 at 10:22

Your example 3 is incorrect! On each system I have tried, the file a.txt will be removed in that scenario. – frnknstn Sep 11 '17 at 10:22

Susam Pal ,Sep 11, 2017 at 15:20

@frnknstn You are right. I see the same behaviour you mention on my latest Debian system. I don't remember on which version of Debian I performed the earlier experiments. In my earlier experiments on an older version of Debian, either a.txt must have survived in the third example or I must have made an error in my experiment. I have updated the answer with the current behaviour I observe on Debian 9 and this behaviour is consistent with what you mention. – Susam Pal Sep 11 '17 at 15:20

Ken Simon ,Jan 25, 2012 at 16:43

Your /home/me/msg directory will be safe if you rm -rf the directory from which you ran ls. Only the symlink itself will be removed, not the directory it points to.

The only thing I would be cautious of, would be if you called something like "rm -rf msg/" (with the trailing slash.) Do not do that because it will remove the directory that msg points to, rather than the msg symlink itself.

> ,Jan 25, 2012 at 16:54

"The only thing I would be cautious of, would be if you called something like "rm -rf msg/" (with the trailing slash.) Do not do that because it will remove the directory that msg points to, rather than the msg symlink itself." - I don't find this to be true. See the third example in my response below. – Susam Pal Jan 25 '12 at 16:54

Andrew Crabb ,Nov 26, 2013 at 21:52

I get the same result as @Susam ('rm -r symlink/' does not delete the target of symlink), which I am pleased about as it would be a very easy mistake to make. – Andrew Crabb Nov 26 '13 at 21:52

,

rm should remove files and directories. If the file is symbolic link, link is removed, not the target. It will not interpret a symbolic link. For example what should be the behavior when deleting 'broken links'- rm exits with 0 not with non-zero to indicate failure

[Oct 05, 2018] Sometimes one extra space makes a big difference

Oct 05, 2018 | cam.ac.uk

From: [email protected] (Richard H. E. Eiger)
Organization: Olivetti (Schweiz) AG, Branch Office Berne

In article <[email protected]> [email protected]
(Tim Smith) writes:
> I was working on a line printer spooler, which lived in /etc. I wanted
> to remove it, and so issued the command "rm /etc/lpspl." There was only
> one problem. Out of habit, I typed "passwd" after "/etc/" and removed
> the password file. Oops.
>
[deleted to save space[
>
> --Tim Smith

Here's another story. Just imagine having the sendmail.cf file in /etc. Now, I was working on the sendmail stuff and had come up with lots of sendmail.cf.xxx which I wanted to get rid of so I typed "rm -f sendmail.cf. *". At first I was surprised about how much time it took to remove some 10 files or so. Hitting the interrupt key, when I finally saw what had happened was way to late, though.

Fortune has it that I'm a very lazy person. That's why I never bothered to just back up directories with data that changes often. Therefore I managed to restore /etc successfully before rebooting... :-) Happy end, after all. Of course I had lost the only well working version of my sendmail.cf...

Richard

[Oct 05, 2018] Deletion of files purpose of which you do not understand sometimes backfire by Anatoly Ivasyuk

Oct 05, 2018 | cam.ac.uk

Unix Admin. Horror Story Summary, version 1.0 by Anatoly Ivasyuk

From: [email protected] (Philip Enteles)
Organization: Haas School of Business, Berkeley

As a new system administrator of a Unix machine with limited space I thought I was doing myself a favor by keeping things neat and clean. One
day as I was 'cleaning up' I removed a file called 'bzero'.

Strange things started to happen like vi didn't work then the compliants started coming in. Mail didn't work. The compilers didn't work. About this time the REAL system administrator poked his head in and asked what I had done.

Further examination showed that bzero is the zeroed memory without which the OS had no operating space so anything using temporary memory was non-functional.

The repair? Well things are tough to do when most of the utilities don't work. Eventually the REAL system administrator took the system to single user and rebuilt the system including full restores from a tape system. The Moral is don't be to anal about things you don't understand.

Take the time learn what those strange files are before removing them and screwing yourself.

Philip Enteles

[Oct 05, 2018] Danger of hidden symlinks

Oct 05, 2018 | cam.ac.uk

From: [email protected] (Chris Calabrese)
Organization: AT&T Bell Labs, Murray Hill, NJ, USA

In article <[email protected]> [email protected] writes:
>On a old decstation 3100

I was deleting last semesters users to try to dig up some disk space, I also deleted some test users at the same time.

One user took longer then usual, so I hit control-c and tried ls. "ls: command not found"

Turns out that the test user had / as the home directory and the remove user script in Ultrix just happily blew away the whole disk.

>U...~

[Oct 05, 2018] Hidden symlinks and recursive deletion of the directories

Notable quotes:

"... Fucking asshole ex-sysadmin taught me a good lesson about checking for symlink bombs. ..."

Oct 05, 2018 | www.reddit.com

mavantix Jack of All Trades, Master of Some; 5 years ago (4 children)

I was cleaning up old temp folders of junk on Windows 2003 server, and C:\temp was full of shit. Most of it junk. Rooted deep in the junk, some asshole admin had apparently symlink'd sysvol to a folder in there. Deleting wiped sysvol.

There where no usable backups, well, there where but the ArcServe was screwed by lack of maintenance.

Spent days rebuilding policies.

Fucking asshole ex-sysadmin taught me a good lesson about checking for symlink bombs.

...and no I didn't tell this story to teach any of your little princesses to do the same when you leave your company.

[Oct 05, 2018] Automatically putting slash in front of directory named like system (named like bin,etc,usr, var) which are all etched in sysadmin memory

This is why you should never type rm command on command line. Type it in editor first.

Oct 05, 2018 | www.reddit.com

aultl Senior DevOps Engineer

rm -rf /var

I was trying to delete /var/named/var

nekoeth0 Linux Admin, 5 years ago

Haha, that happened to me too. I had to use a live distro, chroot, copy, what not. It was fun!

[Oct 02, 2018] Rookie almost wipes customer's entire inventory unbeknownst to sysadmin

Notable quotes:

"... At that moment, everything from / and onward began deleting forcefully and Reginald described his subsequent actions as being akin to "flying flat like a dart in the air, arms stretched out, pointy finger fully extended" towards the power switch on the mini computer. ..."

Oct 02, 2018 | theregister.co.uk

I was going to type rm -rf /*.old* – which would have forcibly removed all /dgux.old stuff, including any sub-directories I may have created with that name," he said.

But – as regular readers will no doubt have guessed – he didn't.

"I fat fingered and typed rm -rf /* – and then I accidentally hit enter instead of the "." key."

At that moment, everything from / and onward began deleting forcefully and Reginald described his subsequent actions as being akin to "flying flat like a dart in the air, arms stretched out, pointy finger fully extended" towards the power switch on the mini computer.

"Everything got quiet."

Reginald tried to boot up the system, but it wouldn't. So instead he booted up off a tape drive to run the mini Unix installer and mounted the boot "/" file system as if he were upgrading – and then checked out the damage.

"Everything down to /dev was deleted, but I was so relieved I hadn't deleted the customer's database and only system files."

Reginald did what all the best accident-prone people do – kept the cock-up to himself, hoped no one would notice and started covering his tracks, by recreating all the system files.

Over the next three hours, he "painstakingly recreated the entire device tree by hand", at which point he could boot the machine properly – "and even the application worked out".

Jubilant at having managed the task, Reginald tried to keep a lid on the heart that was no doubt in his throat by this point and closed off his work, said goodbye to the sysadmin and went home to calm down. Luckily no one was any the wiser.

"If the admins read this message, this would be the first time they hear about it," he said.

"At the time they didn't come in to check what I was doing, and the system was inaccessible to the users due to planned maintenance anyway."

Did you feel the urge to confess to errors no one else at your work knew about? Do you know someone who kept something under their hat for years? Spill the beans to Who, Me? by emailing us here . ® Re: If rm -rf /* doesn't delete anything valuable

Eh? As I read it, Reginald kicked off the rm -rf /*, then hit the power switch before it deleted too much. The tape rescue revealed that "everything down to /dev" had been deleted, ie. everything in / beginnind a,b,c and some d. On a modern system that might include /boot and /bin, but evidently was not a total disaster on Reg's server.

---

Anonymous Coward

title="Inappropriate post? Report it to our moderators" type="submit" value="Report abuse"> I remember discovering the hard way that when you delete an email account in Thunderbird and it asks if you want to delete all the files associated with it it actually means do you want to delete the entire directory tree below where the account is stored .... so, as I discovered, saying "yes" when the reason you are deleting the account is because you'd just created it in the wrong place in the the directory tree is not a good idea - instead of just deleting the new account I nuked all the data associated with all our family email accounts!

div Re: .cobol

"Delete is right above Rename in the bloody menu"

Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

I found this out - to my peril - about 3 seconds before I realised that it was a good idea for a server's crontab to include a daily executed crontab -l > /foo/bar/crontab-backup.txt ...

div Re: .cobol

I went to delete the original files, but I only got as far as "del *.COB" befiore hitting return.

I managed a similar thing but more deliberately; belatedly finding "DEL FOOBAR.???" included files with no extensions when it didn't on a previous version (Win3.1?).

That wasn't the disaster it could have been but I've had my share of all-nighters making it look like I hadn't accidentally scrubbed a system clean.

div Re: .cobol

Probably designed by the same person who designed the crontab app then, with the command line options -e to edit and -r to remove immediately without confirmation. Misstype at your peril...

Using crontab -e is asking for trouble even without mistypes. I've see too many corrupted or truncated crontabs after someone has edited them with crontab -e. crontab -l > crontab.txt;vi crontab.txt;crontab crontab.txt is much better way.

You mean not everyone has crontab entry that backs up crontab at least daily?

div Re: .cobol

"WAH! I copied the .COBOL back to .COB and started over again. As I knew what I wanted to do this time, it only took about a day to re-do what I had deleted."

When this has happened to me, I end up with better code than I had before. Re-doing the work gives you a better perspective. Even if functionally no different it will be cleaner, well commented, and laid out more consistently. I sometimes now do it deliberately (although just saving the first new version, not deleting it) to clean up the code.

div Re: .cobol

I totally agree, the resultant code was better than what I had previously written, because some of the mistakes and assumptions I'd made the first time round and worked around didn't make it into the new code.

div Reminds me of the classic

https://www.ee.ryerson.ca/~elf/hack/recovery.html

div Re: Reminds me of the classic

https://www.ee.ryerson.ca/~elf/hack/recovery.html

Was about to post the same. It is a legendary classic by now.

div One simple trick...

...depending on your shell and its configuration a zero size file in each directory you care about called '-i' will force the rampaging recursive rm, mv, or whatever back into interactive mode. By and large it won't defend you against mistakes in a script, but its definitely saved me from myself when running an interactive shell.

It's proven useful enough to earn its own cronjob that runs once a week and features a 'find -type d' and touch '-i' combo on systems I like.

Glad the OP's mad dive for the power switch saved him, I wasn't so speedy once. Total bustification. Hence this one simple trick...

Now if I could ever fdisk the right f$cking disk, I'd be set!

div Re: One simple trick...

"Can't you enter a command to abort the wipe?"

Maybe. But you still have to work out what got deleted.

On the first Unix system I used, an admin configured the rm command with a system alias so that rm required a confirmation. Annoying after a while but handy when learning.

When you are reconfiguring a system, delete/rm is not the only option. Move/mv protects you from your errors. If the OS has no move/mv, then copy, verify before delete.

div Re: One simple trick...

"Move/mv protects you from your errors."

Not entirely. I had a similar experience with mv. I was left with a running shell so could cd through the remains of the file system end list files with echo * but not repair it..

Although we had the CDs (SCO) to reboot the system required a specific driver which wasn't included on the CDs and hadn't been provided by the vendor. It took most of a day before they emailed the correct driver to put on a floppy before I could reboot. After that it only took a few minutes to put everything back in place.

div Re: One simple trick...

@Chris Evans,

Yes there are a number of things you can do. Just like Windows a quick ctrl-C will abort a rm operation taking place in an interactive shell. Destroying the window in which the interactive shell running rm is running will work, too (alt-f4 in most window managers or 'x' out of the window)

If you know the process id of the rm process you can 'kill $pid' or do a 'killall -KILL rm'

Couple of problems:

(1) law of maximum perversity says that the most important bits will be destroyed first in any accident sequence

(2) by the time you realize the mistake there is no time to kill rm before law 1 is satisfied

The OP's mad dive for the power button is probably the very best move... provided you are right there at the console. And provided the big red switch is actually connected to anything

div cp can also be dangerous

After several years working in a DOS environment I got a job as project Manager / Sys admin on a Unix based customer site for a six month stint. On my second day I wanted to use a test system to learn the software more, so decided to copy the live order files to the test system.

Unfortunately I forgot the trailing full stop as it was not needed in DOS - so the live order index file over wrote the live data file. And the company only took orders for next day delivery so it wiped all current orders.

Luckily it printed a sales acknowledgement every time an order was placed so I escaped death and learned never miss the second parameter with cp command.

title="Inappropriate post? Report it to our moderators" type="submit" value="Report abuse"> i'd written a script to deploy the latest changes to the live environment. worked great. except one day i'd entered a typo and it was now deploying the same files to the remote directory, over and again.

it did that for 2 whole years with around 7 code releases. not a single person realised the production system was running the same code after each release with no change in functionality. all the customer cared about was 'was the site up?'

not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years??? Yet Another Anonymous coward

div Look on the bright side, any bugs your team had introduced in those 2 years had been blocked by your intrinsically secure script

div not a single person realised. not the developers. not the support staff. not me. not the testers. not the customer. just made you think... wtf had we been doing for 2 years???

That is Classic! not surprised about the AC!

Bet some of the beancounters were less than impressed , probly on customer side :)

title="Inappropriate post? Report it to our moderators" type="submit" value="Report abuse"> Re: ...then there's backup stories...

Many years ago (pre internet times) a client phones at 5:30 Friday afternoon. It was the IT guy wanting to run through the steps involved in recovering from a backup. Their US headquarters had a hard disk fail on their accounting system. He was talking the Financial Controller through a recovery and while he knew his stuff he just wanted to double check everything.

8pm the same night the phone rang again - how soon could I fly to the states? Only one of the backup tapes was good. The financial controller had put the sole remaining good backup tape in the drive, then popped out to get a bite to eat at 7pm because it was going to be a late night. At 7:30pm the scheduled backup process copied the corrupted database over the only remaining backup.

Saturday was spent on the phone trying to talk them through everything I could think of.

Sunday afternoon I was sitting in a private jet winging it's way to their US HQ. Three days of very hard work later we'd managed to recreate the accounting database from pieces of corrupted databases and log files. Another private jet ride home - this time the pilot was kind enough to tell me there was a cooler full of beer behind my seat. Olivier2553

div Re: Welcome to the club!

"Lesson learned: NEVER decide to "clean up some old files" at 4:30 on a Friday afternoon. You WILL look for shortcuts and it WILL bite you on the ass."

Do not do anything of some significance on Friday. At all. Any major change, big operation, etc. must be made by Thursday at the latest, so in case of cock-up, you have the Friday (plus days week-end) to repair it.

div I once wiped a large portion of a hard drive after using find with exec rm -rf {} - due to not taking into account the fact that some directories on the system had spaces in them.

div Defensive typing

I've long been in the habit of entering dangerous commands partially in reverse, so in the case of theO/Ps one I've have done:

' -rf /*.old* '

then gone back top the start of the line and entered the ' rm ' bit.

div A couple months ago on my home computer (which has several Linux distros installed and which all share a common /home because I apparently like to make life difficult for myself - and yes, that's as close to a logical reason I have for having multiple distros installed on one machine) I was going to get rid of one of the extraneous Linux installs and use the space to expand the root partition for one of the other distros. I realized I'd typed /dev/sdc2 instead of /dev/sdc3 at the same time that I verified that, yes, I wanted to delete the partition. And sdc2 is where the above mentioned shared /home lives. Doh.

Fortunately I have a good file server and a cron job running rsync every night, so I didn't actually lose any data, but I think my heart stopped for a few seconds before I realized that.

div Came in to work one Monday to find that the Unix system was borked... on investigation it appeared that a large number of files & folders had gone missing, probably by someone doing an incorrect rm.

Our systems were shared with our US office who supported the UK outside of our core hours (we were in from 7am to ensure trading was ready for 8am, they were available to field staff until 10pm UK time) so we suspected it was one of our US counterparts who had done it, but had no way to prove it.

Rather than try and fix anything, they'd gone through and deleted all logs and history entries so we could never find the evidence we needed!

Restoring the system from a recent backup brought everything back online again, as one would expect!

div Sure they did, but the universe invented better idiots

Of course. However, the incompletely-experienced often choose to force bypass that configuration. For example, a lot of systems aliased rm to "rm -i" by default, which would force interactive confirmations. People would then say "UGH, I hate having to do this" and add their own customisations to their shells/profiles etc:

unalias rm

alias rm=rm -f

Lo and behold, now no silly confirmations, regardless of stupidity/typos/etc.

[Apr 22, 2018] THE classic Unix horror story programming

Looks like not much changed since 1986. I amazed how little changed with Unix over the years. RM remains a danger although zsh and -I option on gnu rm are improvement. . I think every sysadmin wiped out important data with rm at least once in his career. So more work on this problem is needed.

Notable quotes:

"... Because we are creatures of habit. If you ALWAYS have to type 'yes' for every single deletion, it will become habitual, and you will start doing it without conscious thought. ..."

"... Amazing what kind of damage you can recover from given enough motivation. ..."

"... " in "rm -rf ~/ ..."

Apr 22, 2008 | www.reddit.com

probablycorey 10 years ago (35 children)

A little trick I use to ensure I never delete the root or home dir... Put a file called -i in / and ~

If you ever call rm -rf *, -i (the request confirmation option) will be the first path expanded. So your command becomes...

rm -rf -i

Catastrophe Averted!

mshade 10 years ago (0 children)

That's a pretty good trick! Unfortunately it doesn't work if you specify the path of course, but will keep you from doing it with a PWD of ~ or /.

Thanks!

aythun 10 years ago (2 children)

Or just use zsh. It's awesome in every possible way.

brian@xenon:~/tmp/test% rm -rf *
zsh: sure you want to delete all the files in /home/brian/tmp/test [yn]?

rex5249 10 years ago (1 child)

I keep an daily clone of my laptop and I usually do some backups in the middle of the day, so if I lose a disk it isn't a big deal other than the time wasted copying files.

MyrddinE 10 years ago (1 child)

Because we are creatures of habit. If you ALWAYS have to type 'yes' for every single deletion, it will become habitual, and you will start doing it without conscious thought.

Warnings must only pop up when there is actual danger, or you will become acclimated to, and cease to see, the warning.

This is exactly the problem with Windows Vista, and why so many people harbor such ill-will towards its 'security' system.

zakk 10 years ago (3 children)

and if I want to delete that file?!? ;-)

alanpost 10 years ago (0 children)

I use the same trick, so either of:

$ rm -- -i

or

$ rm ./-i

will work.

emag 10 years ago (0 children)

rm /-i ~/-i

nasorenga 10 years ago * (2 children)

The part that made me the most nostalgic was his email address: mcvax!ukc!man.cs.ux!miw

Gee whiz, those were the days... (Edit: typo)

floweryleatherboy 10 years ago (6 children)

One of my engineering managers wiped out an important server with rm -rf. Later it turned out he had a giant stock of kiddy porn on company servers.

monstermunch 10 years ago (16 children)

Whenever I use rm -rf, I always make sure to type the full path name in (never just use *) and put the -rf at the end, not after the rm. This means you don't have to worry about hitting "enter" in the middle of typing the path name (it won't delete the directory because the -rf is at the end) and you don't have to worry as much about data deletion from accidentally copy/pasting the command somewhere with middle click or if you redo the command while looking in your bash history.

Hmm, couldn't you alias "rm -rf" to mv the directory/files to a temp directory to be on the safe side?

branston 10 years ago (8 children)

Aliasing 'rm' is fairly common practice in some circles. It can have its own set of problems however (filling up partitions, running out of inodes...)

amnezia 10 years ago (5 children)

you could alias it with a script that prevents rm -rf * being run in certain directories.

jemminger 10 years ago (4 children)

you could also alias it to 'ls' :)

derefr 10 years ago * (1 child)

One could write a daemon that lets the oldest files in that directory be "garbage collected" when those conditions are approaching. I think this is, in a roundabout way, how Windows' shadow copy works.

branston 10 years ago (0 children)

Could do. Think we might be walking into the over-complexity trap however. The only time I've ever had an rm related disaster was when accidentally repeating an rm that was already in my command buffer. I looked at trying to exclude patterns from the command history but csh doesn't seem to support doing that so I gave up.

A decent solution just occurred to me when the underlying file system supports snapshots (UFS2 for example). Just snap the fs on which the to-be-deleted items are on prior to the delete. That needs barely any IO to do and you can set the snapshots to expire after 10 minutes.

Hmm... Might look at implementing that..

mbm 10 years ago (0 children)

Most of the original UNIX tools took the arguments in strict order, requiring that the options came first; you can even see this on some modern *BSD systems.

shadowsurge 10 years ago (1 child)

I just always format the command with ls first just to make sure everything is in working order. Then my neurosis kicks in and I do it again... and a couple more times just to make sure nothing bad happens.

Jonathan_the_Nerd 10 years ago (0 children)

If you're unsure about your wildcards, you can use echo to see exactly how the shell will expand your arguments.

splidge 10 years ago (0 children)

A better trick IMO is to use ls on the directory first.. then when you are sure that's what you meant type rm -rf !$ to delete it.

earthboundkid 10 years ago * (0 children)

Ever since I got burned by letting my pinky slip on the enter key years ago, I've been typing echo path first, then going back and adding the rm after the fact.

zerokey 10 years ago * (2 children)

Great story. Halfway through reading, I had a major wtf moment. I wasn't surprised by the use of a VAX, as my old department just retired their last VAX a year ago. The whole time, I'm thinking, "hello..mount the tape hardware on another system and, worst case scenario, boot from a live cd!"

Then I got to, "The next idea was to write a program to make a device descriptor for the tape deck" and looked back at the title and realized that it was from 1986 and realized, "oh..oh yeah...that's pretty fucked."

iluvatar 10 years ago (0 children)

Great story

Yeah, but really, he had way too much of a working system to qualify for true geek godhood. That title belongs to Al Viro . Even though I've read it several times, I'm still in awe every time I see that story...

cdesignproponentsist 10 years ago (0 children)

FreeBSD has backup statically-linked copies of essential system recovery tools in /rescue, just in case you toast /bin, /sbin, /lib, ld-elf.so.1, etc.

It won't protect against a rm -rf / though (and is not intended to), although you could chflags -R schg /rescue to make them immune to rm -rf.

clytle374 10 years ago * (9 children)

It happens, I tried a few months back to rm -rf bin to delete a directory and did a rm -rf /bin instead.

First thought: That took a long time.

Second thought: What do you mean ls not found.

I was amazed that the desktop survived for nearly an hour before crashing.

earthboundkid 10 years ago (8 children)

This really is a situation where GUIs are better than CLIs. There's nothing like the visual confirmation of seeing what you're obliterating to set your heart into the pit of your stomach.

jib 10 years ago (0 children)

If you're using a GUI, you probably already have that. If you're using a command line, use mv instead of rm.

In general, if you want the computer to do something, tell it what you want it to do, rather than telling it to do something you don't want and then complaining when it does what you say.

earthboundkid 10 years ago (3 children)

Yes, but trash cans aren't manly enough for vi and emacs users to take seriously. If it made sense and kept you from shooting yourself in the foot, it wouldn't be in the Unix tradition.

earthboundkid 10 years ago (1 child)

1. Are you so low on disk space that it's important for your trash can to be empty at all times?
2. Why should we humans have to adapt our directory names to route around the busted-ass-ness of our tools? The tools should be made to work with capital letters and spaces. Or better, use a GUI for deleting so that you don't have to worry about OMG, I forgot to put a slash in front of my space!

Seriously, I use the command line multiple times every day, but there are some tasks for which it is just not well suited compared to a GUI, and (bizarrely considering it's one thing the CLI is most used for) one of them is moving around and deleting files.

easytiger 10 years ago (0 children)

Thats a very simple bash/ksh/python/etc script.

1. script a move op to a hidden dir on the /current/ partition.
2. alias this to rm
3. wrap rm as an alias to delete the contents of the hidden folder with confirmation

mattucf 10 years ago (3 children)

I'd like to think that most systems these days don't have / set as root's home directory, but I've seen a few that do. :/

dsfox 10 years ago (0 children)

This is a good approach in 1986. Today I would just pop in a bootable CDROM.

fjhqjv 10 years ago * (5 children)

That's why I always keep stringent file permissions and never act as the root user.

I'd have to try to rm -rf, get a permission denied error, then retype sudo rm -rf and then type in my password to ever have a mistake like that happen.

But I'm not a systems administrator, so maybe it's not the same thing.

toast_and_oj 10 years ago (2 children)

I aliased "rm -rf" to "omnomnom" and got myself into the habit of using that. I can barely type "omnomnom" when I really want to, let alone when I'm not really paying attention. It's saved one of my projects once already.

shen 10 years ago (0 children)

I've aliased "rm -rf" to "rmrf". Maybe I'm just a sucker for punishment.

I haven't been bit by it yet, the defining word being yet.

robreim 10 years ago (0 children)

I would have thought tab completion would have made omnomnom potentially easier to type than rm -rf (since the -rf part needs to be typed explicitly)

immure 10 years ago (0 children)

It's not.

lespea 10 years ago (0 children)

before I ever do something like that I make sure I don't have permissions so I get an error, then I press up, home, and type sudo <space> <enter> and it works as expected :)

kirun 10 years ago (0 children)

And I was pleased the other day how easy it was to fix the system after I accidentally removed kdm, konqueror and kdesktop... but these guys are hardcore.

austin_k 10 years ago (0 children)

I actually started to feel sick reading that. I've been in a IT disaster before where we almost lost a huge database. Ugh.. I still have nightmares.

umilmi81 10 years ago (4 children)

Task number 1 with a UNIX system. Alias rm to rm -i. Call the explicit path when you want to avoid the -i (ie: /bin/rm -f). Nobody is too cool to skip this basic protection.

flinchn 10 years ago (0 children)

i did an application install at an LE agency last fall - stupid me mv ./etc ./etcbk <> mv /etc /etcbk

ahh that damned period

DrunkenAsshole 10 years ago (0 children)

Were the "*"s really needed for a story that has plagued, at one point or another, all OS users?

xelfer 10 years ago (0 children)

Is the home directory for root / for some unix systems? i thought 'cd' then 'rm -rf *' would have deleted whatever's in his home directory (or whatever $HOME points to)

srparish 10 years ago (0 children)

Couldn't he just have used the editor to create the etc files he wanted, and used cpio as root to copy that over as an /etc?

sRp

stox 10 years ago (1 child)

Been there, done that. Have the soiled underwear to prove it. Amazing what kind of damage you can recover from given enough motivation.

sheepskin 10 years ago * (0 children)

I had a customer do this, he killed it about the same time. I told him he was screwed and I'd charge him a bunch of money to take down his server, rebuild it from a working one and put it back up. But the customer happened to have a root ftp session up, and was able to upload what he needed to bring the system back. by the time he was done I rebooted it to make sure it was cool and it booted all the way back up.

Of course I've also had a lot of customer that have done it, and they where screwed, and I got to charge them a bunch of money.

jemminger 10 years ago (0 children)

pfft. that's why lusers don't get root access.

supersan 10 years ago (2 children)

i had the same thing happened to me once.. my c:\ drive was running ntfs and i accidently deleted the "ntldr" system file in the c:\ root (because the name didn't figure much).. then later, i couldn't even boot in the safe mode! and my bootable disk didn't recognize the c:\ drive because it was ntfs!! so sadly, i had to reinstall everything :( wasted a whole day over it..

b100dian 10 years ago (0 children)

Yes, but that's a single file. I suppose anyone can write hex into mbr to copy ntldr from a samba share!

bobcat 10 years ago (0 children)

http://en.wikipedia.org/wiki/Emergency_Repair_Disk

boredzo 10 years ago (0 children)

Neither one is the original source. The original source is Usenet, and I can't find it with Google Groups. So either of these webpages is as good as the other.

docgnome 10 years ago (0 children)

In 1986? On a VAX?

MarlonBain 10 years ago (0 children)

This classic article from Mario Wolczko first appeared on Usenet in 1986 .

amoore 10 years ago (0 children)

I got sidetracked trying to figure out why the fictional antagonist would type the extra "/ " in "rm -rf ~/ ".

Zombine 10 years ago (2 children)

...it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal.

Yeah. So apart from the fact that no one could get any work done or really do anything, things were working great!

I think a more rational reaction would be "Why on Earth is this big, important system on which many people rely designed in such a way that a simple easy-to-make human error can screw it up so comprehensively?" or perhaps "Why on Earth don't we have a proper backup system?"

daniels220 10 years ago (1 child)

The problem wasn't the backup system, it was the restore system, which relied on the machine having a "copy" command. Perfectly reasonable assumption that happened not to be true.

Zombine 10 years ago * (0 children)

Neither backup nor restoration serves any purpose in isolation. Most people would group those operations together under the heading "backup;" certainly you win only a semantic victory by doing otherwise. Their fail-safe data-protection system, call it what you will, turned out not to work, and had to be re-engineered on-the-fly.

I generally figure that the assumptions I make that turn out to be entirely wrong were not "perfectly reasonable" assumptions in the first place. Call me a traditionalist.

[Apr 22, 2018] rm and Its Dangers (Unix Power Tools, 3rd Edition)

Apr 22, 2018 | docstore.mik.ua

14.3. rm and Its Dangers

Under Unix, you use the rm command to delete files. The command is simple enough; you just type rm followed by a list of files. If anything, rm is too simple. It's easy to delete more than you want, and once something is gone, it's permanently gone. There are a few hacks that make rm somewhat safer, and we'll get to those momentarily. But first, here's a quick look at some of the dangers.

To understand why it's impossible to reclaim deleted files, you need to know a bit about how the Unix filesystem works. The system contains a "free list," which is a list of disk blocks that aren't used. When you delete a file, its directory entry (which gives it its name) is removed. If there are no more links ( Section 10.3 ) to the file (i.e., if the file only had one name), its inode ( Section 14.2 ) is added to the list of free inodes, and its datablocks are added to the free list.

Well, why can't you get the file back from the free list? After all, there are DOS utilities that can reclaim deleted files by doing something similar. Remember, though, Unix is a multitasking operating system. Even if you think your system is a single-user system, there are a lot of things going on "behind your back": daemons are writing to log files, handling network connections, processing electronic mail, and so on. You could theoretically reclaim a file if you could "freeze" the filesystem the instant your file was deleted -- but that's not possible. With Unix, everything is always active. By the time you realize you made a mistake, your file's data blocks may well have been reused for something else.

When you're deleting files, it's important to use wildcards carefully. Simple typing errors can have disastrous consequences. Let's say you want to delete all your object ( .o ) files. You want to type:

% rm *.o

But because of a nervous twitch, you add an extra space and type:

% rm * .o

It looks right, and you might not even notice the error. But before you know it, all the files in the current directory will be gone, irretrievably.

If you don't think this can happen to you, here's something that actually did happen to me. At one point, when I was a relatively new Unix user, I was working on my company's business plan. The executives thought, so as to be "secure," that they'd set a business plan's permissions so you had to be root ( Section 1.18 ) to modify it. (A mistake in its own right, but that's another story.) I was using a terminal I wasn't familiar with and accidentally created a bunch of files with four control characters at the beginning of their name. To get rid of these, I typed (as root ):

# rm ????*

This command took a long time to execute. When about two-thirds of the directory was gone, I realized (with horror) what was happening: I was deleting all files with four or more characters in the filename.

The story got worse. They hadn't made a backup in about five months. (By the way, this article should give you plenty of reasons for making regular backups ( Section 38.3 ).) By the time I had restored the files I had deleted (a several-hour process in itself; this was on an ancient version of Unix with a horrible backup utility) and checked (by hand) all the files against our printed copy of the business plan, I had resolved to be very careful with my rm commands.

[Some shells have safeguards that work against Mike's first disastrous example -- but not the second one. Automatic safeguards like these can become a crutch, though . . . when you use another shell temporarily and don't have them, or when you type an expression like Mike's very destructive second example. I agree with his simple advice: check your rm commands carefully! -- JP ]

-- ML

[Apr 22, 2018] How to prevent a mistaken rm -rf for specific folders?

Notable quotes:

"... There's nothing more on a traditional Linux, but you can set Apparmor/SELinux/ rules that prevent rm from accessing certain directories. ..."

"... Probably your best bet with it would be to alias rm -ri into something memorable like kill_it_with_fire . This way whenever you feel like removing something, go ahead and kill it with fire. ..."

Jan 20, 2013 | unix.stackexchange.com

I think pretty much people here mistakenly 'rm -rf'ed the wrong directory, and hopefully it did not cause a huge damage.. Is there any way to prevent users from doing a similar unix horror story?? Someone mentioned (in the comments section of the previous link) that

... I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it ...

Is there any implementation of that in any current Unix or Linux distro? And what is the common practice to prevent that error even from a sysadmin (with root access)?

It seems that there was some protection for the root directory (/) in Solaris (since 2005) and GNU (since 2006). Is there anyway to implement the same protection way to some other folders as well??

To give it more clarity, I was not asking about general advice about rm usage (and I've updated the title to indicate that more), I want something more like the root folder protection: in order to rm -rf / you have to pass a specific parameter: rm -rf --no-preserve-root /.. Is there similar implementations for customized set of directories? Or can I specify files in addition to / to be protected by the preserve-root option?

---

amyassin, Jan 20, 2013 at 17:26

I think pretty much people here mistakenly ' rm -rf 'ed the wrong directory, and hopefully it did not cause a huge damage.. Is there any way to prevent users from doing a similar unix horror story ?? Someone mentioned (in the comments section of the previous link ) that

... I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it ...

Is there any implementation of that in any current Unix or Linux distro? And what is the common practice to prevent that error even from a sysadmin (with root access)?

It seems that there was some protection for the root directory ( / ) in Solaris (since 2005) and GNU (since 2006). Is there anyway to implement the same protection way to some other folders as well??

To give it more clarity, I was not asking about general advice about rm usage (and I've updated the title to indicate that more), I want something more like the root folder protection: in order to rm -rf / you have to pass a specific parameter: rm -rf --no-preserve-root / .. Is there similar implementations for customized set of directories? Or can I specify files in addition to / to be protected by the preserve-root option?

mattdm, Jan 20, 2013 at 17:33

1) Change management 2) Backups. – mattdm Jan 20 '13 at 17:33

Keith, Jan 20, 2013 at 17:40

probably the only way would be to replace the rm command with one that doesn't have that feature. – Keith Jan 20 '13 at 17:40

sr_, Jan 20, 2013 at 18:28

safe-rm maybe – sr_ Jan 20 '13 at 18:28

Bananguin, Jan 20, 2013 at 21:07

most distros do `alias rm='rm -i' which makes rm ask you if you are sure.

Besides that: know what you are doing. only become root if necessary. for any user with root privileges security of any kind must be implemented in and by the user. hire somebody if you can't do it yourself.over time any countermeasure becomes equivalaent to the alias line above if you cant wrap your own head around the problem. – Bananguin Jan 20 '13 at 21:07

midnightsteel, Jan 22, 2013 at 14:21

@amyassin using rm -rf can be a resume generating event. Check and triple check before executing it – midnightsteel Jan 22 '13 at 14:21

Gilles, Jan 22, 2013 at 0:18

To avoid a mistaken rm -rf, do not type rm -rf .

If you need to delete a directory tree, I recommend the following workflow:

• If necessary, change to the parent of the directory you want to delete.
• mv directory-to-delete DELETE
• Explore DELETE and check that it is indeed what you wanted to delete
• rm -rf DELETE

Never call rm -rf with an argument other than DELETE . Doing the deletion in several stages gives you an opportunity to verify that you aren't deleting the wrong thing, either because of a typo (as in rm -rf /foo /bar instead of rm -rf /foo/bar ) or because of a braino (oops, no, I meant to delete foo.old and keep foo.new ).

If your problem is that you can't trust others not to type rm -rf, consider removing their admin privileges. There's a lot more that can go wrong than rm .

---

Always make backups .

Periodically verify that your backups are working and up-to-date.

Keep everything that can't be easily downloaded from somewhere under version control.

---

With a basic unix system, if you really want to make some directories undeletable by rm, replace (or better shadow) rm by a custom script that rejects certain arguments. Or by hg rm .

Some unix variants offer more possibilities.

• On OSX, you can set an access control list on a directory preventing deletion of the files and subdirectories inside it, without preventing the creation of new entries or modification of existing entries: chmod +a 'group:everyone deny delete_child' somedir (this doesn't prevent the deletion of files in subdirectories: if you want that, set the ACL on the subdirectory as well).
• On Linux, you can set rules in SELinux, AppArmor or other security frameworks that forbid rm to modify certain directories.

amyassin, Jan 22, 2013 at 9:41

Yeah backing up is the most amazing solution, but I was thinking of something like the --no-preserve-root option, for other important folder.. And that apparently does not exist even as a practice... – amyassin Jan 22 '13 at 9:41

Gilles, Jan 22, 2013 at 20:32

@amyassin I'm afraid there's nothing more (at least not on Linux). rm -rf already means "delete this, yes I'm sure I know what I'm doing". If you want more, replace rm by a script that refuses to delete certain directories. – Gilles Jan 22 '13 at 20:32

Gilles, Jan 22, 2013 at 22:17

@amyassin Actually, I take this back. There's nothing more on a traditional Linux, but you can set Apparmor/SELinux/ rules that prevent rm from accessing certain directories. Also, since your question isn't only about Linux, I should have mentioned OSX, which has something a bit like what you want. – Gilles Jan 22 '13 at 22:17

qbi, Jan 22, 2013 at 21:29

If you are using rm * and the zsh, you can set the option rmstarwait :

setopt rmstarwait

Now the shell warns when you're using the * :

> zsh -f
> setopt rmstarwait
> touch a b c
> rm *
zsh: sure you want to delete all the files in /home/unixuser [yn]? _

When you reject it ( n ), nothing happens. Otherwise all files will be deleted.

Drake Clarris, Jan 22, 2013 at 14:11

EDIT as suggested by comment:

You can change the attribute of to immutable the file or directory and then it cannot be deleted even by root until the attribute is removed.

chattr +i /some/important/file

This also means that the file cannot be written to or changed in anyway, even by root . Another attribute apparently available that I haven't used myself is the append attribute ( chattr +a /some/important/file . Then the file can only be opened in append mode, meaning no deletion as well, but you can add to it (say a log file). This means you won't be able to edit it in vim for example, but you can do echo 'this adds a line' >> /some/important/file . Using > instead of >> will fail.

These attributes can be unset using a minus sign, i.e. chattr -i file

Otherwise, if this is not suitable, one thing I practice is to always ls /some/dir first, and then instead of retyping the command, press up arrow CTL-A, then delete the ls and type in my rm -rf if I need it. Not perfect, but by looking at the results of ls, you know before hand if it is what you wanted.

NlightNFotis, Jan 22, 2013 at 8:27

One possible choice is to stop using rm -rf and start using rm -ri . The extra i parameter there is to make sure that it asks if you are sure you want to delete the file.

Probably your best bet with it would be to alias rm -ri into something memorable like kill_it_with_fire . This way whenever you feel like removing something, go ahead and kill it with fire.

amyassin, Jan 22, 2013 at 14:24

I like the name, but isn't f is the exact opposite of i option?? I tried it and worked though... – amyassin Jan 22 '13 at 14:24

NlightNFotis, Jan 22, 2013 at 16:09

@amyassin Yes it is. For some strange kind of fashion, I thought I only had r in there. Just fixed it. – NlightNFotis Jan 22 '13 at 16:09

Silverrocker, Jan 22, 2013 at 14:46

To protect against an accidental rm -rf * in a directory, create a file called "-i" (you can do this with emacs or some other program) in that directory. The shell will try to interpret -i and will cause it to go into interactive mode.

For example: You have a directory called rmtest with the file named -i inside. If you try to rm everything inside the directory, rm will first get -i passed to it and will go into interactive mode. If you put such a file inside the directories you would like to have some protection on, it might help.

Note that this is ineffective against rm -rf rmtest .

ValeriRangelov, Dec 21, 2014 at 3:03

If you understand C programming language, I think it is possible to rewrite the rm source code and make a little patch for kernel. I saw this on one server and it was impossible to delete some important directories and when you type 'rm -rf /direcotyr' it send email to sysadmin.

bash - Safe Directory Delete

Mar 13, 2013 | Stack Exchange

rm -i would get extremely annoying especially when removing recursively.

rm -I, on the other hand seems a little more tolerable which will prompt once before removing more than three files, or when removing recursively.

It's much less intrusive while still giving protection against most mistakes.

The best form of protection is to double/triple check when deleting recursively or better yet, anything you delete.

• In my experience, this kind of aliases just makes the "rm foo<ENTER>y<ENTER>" automatic... i.e., no protection at the cost of 2 extra keystrokes. And it isn't available everywhere. Better don't do that, except for the greenest training-wheel-needy users. – vonbrand Mar 13 '13 at 20:20

• Seeing that rm foo would only remove one file. rm foo would only ask once. If you're talking about when removing the directory foo/ then the command rm -Ir foo would ask once as well (if it contained more than 3 files). If you were to remove a directory recursively that had 1000 files in it (for arguments sake). Then you'd be there for quite a bit of time saying Yes to confirm all the files with -i where since the user should know what files their deleting -I will ask once for any removal operation with three or more files involved. – cinelli Mar 13 '13 at 22:41

• Adding to that. If the user doesn't know what's contained in foo/ then when rm -Ir foo is passed and it asks "You sure?" and the user says Yes without double checking and not knowing what's in the directory. Then, that's something nobody can answer other than. "Pay attention to what you're doing to you're system. Or go back to using the Recycle Bin in Windows." – cinelli Mar 13 '13 at 22:47

You can cd to the top level directory you want to remove recursively, then go up a level and rm that directory. That way you don't have to use leading slashes, etc.

cd /home/user/junk
ls -1

folders
to
delete

rm -ri . ###Or "rm -rf ."
cd ..
rm -rf junk

Another way to check that your paths are right for any command is to simply use tab completion.

One option is to use a trash can approach, like here.

There are a lot of examples out there besides this one - it was just the first in my search results. Basically, all rm operations become mv commands. If you rm the wrong files, they're still on the machine and easily recoverable. And the approach linked above creates an alias to the base rm command so you can still use it when you really want to.

[Apr 22, 2018] bash - Is it safe to alias rm to safe-rm

Jun 7, 2014 | Ask Ubuntu

Q:

We are happy to use safe-rm as a replacement to rm, but is it considered safe to alias it?

e.g.

alias rm='safe-rm'

A: Yes, an alias is just a shell-only rename. The scope of an alias is only the shell that you are currently in. safe-rm is pDELETE. Doing the deletion in several stages gives you an opportunity to verify that you aren't deleting the wrong thing, either because of a typo (as in rm -rf /foo /bar instead of rm -rf /foo/bar) or because of a braino (oops, no, I meant to delete foo.old and keep foo.new).

If your problem is that you can't trust others not to type rm -rf, consider removing their admin privileges. There's a lot more that can go wrong than rm.

Let me get into the problem you asked

This tool, safe-rm, ships with a default system-wide configuration file (/etc/safe-rm.conf), but each user can supplement that list of "protected" directories and files by adding lines to ~/.safe-rm.

Use \rm when you are sure about what you are deleting but most of the time, I prefer to go through the confirmation. Too many files disappeared because of a rushed removal action.

THE classic Unix horror story WSU Linux Users Group

*THE* classic Unix horror story

Tue, 01/31/2006 - 10:11am - binford2k

Try this in windows!

---

Unix Recovery Legend

This classic article from Mario Wolczko first appeared on Usenet in 1986.

Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters....

It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg. My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed

grep 147 /etc/passwd

only to receive the response

/etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss. This was confirmed when in response to

ls /etc

I got

ls: not found.

I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager.

When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines:

# cd
# rm -rf *

Oh, shit, I thought. That would just about explain it.

I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail.thing I remte> cd /
echo *

I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know[work?] without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck.

Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable.

So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference:

I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved.

So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history.

Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one.

The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat.

And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together.

I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to:

Mario Wolczko
------------------------------------------------------------------------
Dept. of Computer Science       ARPA:   miw%[email protected]
The University                  USENET: mcvax!ukc!man.cs.ux!miw
Manchester M13 9PL              JANET:                                           [email protected]
U.K.                            061-273 7121 x 5699
------------------------------------------------------------------------

We used to outsource some of our support at our data center, and we had multiple rm's from those agents. Luckily, live CD's can get you back rolling to where you can rsync from backups.

# cd
# rm -rf *

If I'm not mistaken, wouldn't this just delete everything in /root ? I'm more of a Linux guy, but I'd presume you'd need the following command to wipe everything.

rm -rf /

-Chris | Editor
Mozy Reviews

Mon, 04/25/2011 - 3:43pm - binford2k

It all depends on where root's home dir is. Some unixes do have it set to /

Thu, 05/19/2011 - 9:36am - mchristenson

It doesn't matter whether you keep anything in /root or /home, if the distribution you are using has the root user's HOME environment variable set to /.

POSIX standard says that, when not supplied with any arguments, the cd command will change to the directory pointed to by HOME (or an implementation based location if HOME is unset or undefined.) If the former, subsequently invoking rm -rf * would then remove both your /root and /home directories.

See http://pubs.opengroup.org/onlinepubs/009695399/utilities/cd.html

Mon, 01/03/2011 - 12:39am - scott.allerdice

Not classic horrible yes, the admins did a good job of killing the process before it made it's way to the rest of the file system. I had a similar problem on windows in the 90's when malware didn't exist and viruses weren't bots but destroyed files. I went to start run telnet and it started searching for telnet.exe i knew something was wrong so went to my windows folder to look for it manually and didn't see it. How strange i thought i just saw a file disappear and another etc. Yup files were disappearing before my very eyes. It was as bad as rm -rf and i had no time to look and see what was doing it so i pulled the plug on my machine. It's what i would do had i seen rm -rf. Pull the power and figure out a plan. For windows i had norton on floppy and was able to clean it out i don't know what i lost but got the system back up and running. Unix wise i've ran redhat, slack, suse, ubuntu, freebsd, and sparc x86 i make it a point to have weekly backups of my drives and with out using recovery CD's or programs i do it almost manually copying files over to the drives so i know they are in a format i can access once recovered. If a friend ever threatened an rm -rf for $5 i would comply however his next trip home would be memorable i would fill his suite case with sex toys and synthetic marijuana and coke powder, it's legal and used to train drug sniffing dogs. It would make for a hell of a trip through TSA screening and possibly a trip to jail court or maybe even drug rehab i'd tell him he can add the $5 to the bail money i'd put up for him.

Heh, ingenious and creative. I would have given up about half way through (if that)

This is without a doubt the most convoluted restore process I've ever heard of. It sounds to me like the operators involved were too naïve when it came to the install process for 4.3BSD. It really isn't that involved. Honestly, you have to wonder who installed their OS, and why didn't they have access to the manuals? I mean even if they paid for the tape only, wouldn't they have printed out the system manuals?

I could have had that box up & running again in under an hour. First thing to do would be to locate the install tape, and halt the machine. Boot from tape, and run the copy program to install the miniroot into the swap partition. Rebooting the VAX again, into the swap partition I would restore the root partition from one week old root backup.

One more reboot, and the VAX would have it's root back from their best backup, then it would be a matter of restoring other directories as needed.

Oh and yes by default 4.3BSD has separate root partition from the rest of the userland stuff.

I can only think that these people were absolutely terrified of rebooting their VAX. I wonder if it was an 11/780 with a borked console tape/floppy.

But who knows this is over 20 years ago.

One way to avoid the assembler is to use a combination of tar and uuencode

Basically, if you transfer a tar file with only one directory in it (by manual typing) - you would be able to extract it using

tar xvf etc.tar -C /

Only issue is that the size is a bit larger - but I think if one gives enough thought to this - it could be reduced.

The key to this ofcourse - as you say, is to NOT PANIC.

Fri, 03/14/2008 - 3:11pm - joeblow1470

tar is located at /bin/tar as it states in the story(about 1/3 of the way down)(bin was one of the deleted directories)

PS great story

Thu, 10/07/2010 - 10:51am - jfwfmt

not on Unix, on RT-11 but a close equivalent horror story:

One night I was finishing up the code for the VAX 8600 console. I had finally made the changes to the OS code and to the clock driver to translate the 64Hz interrupt rate to the 60Hz clock on the PDP-11 (with a little studder every 1/4 second) and to transfer the 1KHz clock through the mismatched registers windows to the VAX.

So everything was good and i decided to clean up and leave. I entered:

. delete *,tmp

It took a long time and then came back with:

tmp file not found.

Of course the comma rather than the intended dot made all the difference.

. directory

returned with nothing.

Not to worry, there were men in white coats who religiously took backups of everything every night,

Except they had never done a restore.

Which of course failed.

So I came in the next day and started over.

Fri, 12/10/2010 - 10:00pm - greffedecheveux (not verified)

[Apr 21, 2018] Any alias of rm is a very stupid idea

Option -I is more modern and more useful then old option -i. It is highly recommended. And it make sense to to use alias with it contrary to what this author states (he probably does not understand that aliases do not wok for non-interactive sessions.).

The point the author make is that when you automatically expect rm to be aisles to rm -i you get into trouble on machines where this is not the case. And that's completely true.

But it does not solve the problem as respondents soon became automatic. stated. Writing your own wrapper is a better deal. One such wrapper -- safe-rm already exists and while not perfect is useful

Notable quotes:

"... A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do. ..."

"... It you want a safety net, do "alias del='rm -I –preserve_root'", ..."

Feb 14, 2017 | www.cyberciti.biz

Art Protin June 12, 2012, 9:53 pm

Any alias of rm is a very stupid idea (except maybe alias rm=echo fool).

A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do.

It you want a safety net, do "alias del='rm -I –preserve_root'",

Drew Hammond March 26, 2014, 7:41 pm

^ This x10000.

I've made the same mistake before and its horrible.

[Jul 20, 2017] The ULTIMATE Horrors story with recovery!

Notable quotes:

"... Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters.... ..."

"... I can't remember what happened in the succeeding minutes; my memory is just a blur. ..."

"... (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). ..."

"... By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working. ..."

Nov 08, 2002 | www.linuxjournal.com

Anonymous on Fri, 11/08/2002 - 03:00.

Its here .. Unbeliveable..

[I had intended to leave the discussion of "rm -r *" behind after the compendium I sent earlier, but I couldn't resist this one.

I also received a response from rutgers!seismo!hadron!jsdy (Joseph S. D. Yao) that described building a list of "dangerous" commands into a shell and dropping into a query when a glob turns up. They built it in so it couldn't be removed, like an alias. Anyway, on to the story! RWH.] I didn't see the message that opened up the discussion on rm, but thought you might like to read this sorry tale about the perils of rm....

(It was posted to net.unix some time ago, but I think our postnews didn't send it as far as it should have!)

----------------------------------------------------------------

Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters....

It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg.

My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed

grep 147 /etc/passwd

only to receive the response

/etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss. This was confirmed when in response to

ls /etc

I got

ls: not found.

I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager. When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines:

# cd 
# rm -rf * 

Oh, *****, I thought. That would just about explain it.

I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and typing

cd / 
echo * 

I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live?

You guessed it, /etc.

How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck.

Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday).

Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable.

So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in.

Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference:

070100002c000000000000000000000000000000000000000000000000000000 
0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f 
8800040000bc012f65746300 

I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved.

So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all.

From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history.

Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one.

The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat. And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together.

I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way?

Answers on a postage stamp to:

Mario Wolczko

------------------------------------------------------------------------

Dept. of Computer Science ARPA: miw%[email protected]

The University USENET: mcvax!ukc!man.cs.ux!miw

Manchester M13 9PL JANET: [email protected]

U.K. 061-273 7121 x 5699

[Jun 12, 2010] Sysadmin Blunder (3rd Response)

Toolbox for IT Groups

jxtmills:

I needed to clear a directory full of ".directories" and I issued:

rm -r .*

It seemed to find ".." awfully fast. That was a bad day, but, we restored most of the files in about an hour.

John T. Mills

alain:

hi every one

for me , i remember 2

1 - 'rm *;old' in / directory note ';' instead of '.'

2 - kill #pid of the informix process (oninit) and delete it (i dreamed)

jturner:

Variation on a theme ... the 'rm -r theme'

as a junior admin on AIX3.2.5, I had been left to my own devices to create
some housekeeping scripts - all my draft scripts being created and tested
in my home directory with a '.jt' suffix. After completing the scripts I
found that I had inadvertantly placed some copies in / with a .jt suffix.
Easy job then to issue a 'rm *.jt' in / and all would be well. Well it
would have been if I hadn't put a space between the * and the .jt. And the
worst thing of all, not being a touch typist and looking at the keys, I
glanced at the screen before hitting the enter key and realised with horror
what was going to happen and STILL my little finger continued to proceed
toward the enter key.

Talk about 'Gone in 60 seconds' - my life was at an end - over - finished
- perhaps I could park cars or pump gas for a living. Like other
correspondents a helpful senior admin was on hand to smile kindly and show
me how to restore from mksysb-fortunately taken daily on these production
systems. (Thanks Pete :))) )

To this day, rm -i is my first choice with multiple rm's just as a test!!!!!!

Happy rm-ing :)

daguenet:

I know that one. Does any body remember when the rm man page had a warning not do rm -rf / as root? How may systems were rebuild due that blunder. Not that I have ever done something like that, nor will ever admit to it:).

Aaron

cal.staples:

That is a no brainer!

First a little background. I cooked up a script called "killme" which
would ask for a search string then parse the process table and return a
list of all matches. If the list contained the processes you wanted to
kill then you would answer "Yes", not once, but twice just to be sure
you thought about it. This was very handy at times so I put it out on
all of our servers.

Some time had passed and I had not used it for a while when I had a need
to kill a group of processes. So I typed the command not realizing that
I had forgotten the scripts name. Of course I was on our biggest
production system at that time and everything stopped in it's tracks!

Unknown to me was that there is an AIX command called "killall" which is
what I typed.

From the MAN page: "The killall command cancels all processes that you
started, except those producing the killall process. This command
provides a convenient means of canceling all processes created by the
shell that you control. When started by a root user, the killall command
cancels all cancellable processes except those processes that started
it."

And it doesn't ask for confirmation or anything! Fortunately the
database didn't get corrupted and we were able to bring everything back
on line fairly quickly. Needless to say we changed the name of this
command so it couldn't be run so easily.

"Killall" is a great command for a programmer developing an application
which goes wild and he/she needs to kill the processes and retain
control of the system, but it is very dangerous in the real world!

Jeff Scott:

The silliest mistake? That had to be a permissions change on /bin. I got a
call from an Oracle DBA that the $ORACLE_HOME/bin no longer belonged to
oracle:dba. We never found out how that happened. I logged in to change the
permissions. I accidentally typed cd /oracle.... /bin (note the space
before /bin), then cheerfully entered the following command:

#chown -R oracle:dba ./*

The command did not climb up to root fortunately, but it really made a mess.
We ended up restoring /bin from a backup taken the previous evening.

Jeff Scott
Darwin Partners/EMC

Cal S.

tzhou:

crontab -r when I wanted to do crontab -e. See letters e and r are side by
side on the keyboard. I had 2 pages of crontab and had no backup on the
machine !

Jeff Scott:

I've seen the rm -fr effect before. There were no entries in any crontab.
Before switching to sudo, the company used a homegrown utility to grant
things root access. The server accepted ETLs from other databases, acting as
a data warehouse. This utility logged locally, instead of logging via
syslogd with local and remote logging. So, when the system was erased, there
really was no way to determine the actual cause. Most of the interactive
users shared a set of group accounts, instead of enforcing individual
accounts and using su - or sudo. The outage cost the company $8 million USD
due to lost labor that had to be repeated. Clearly, it was caused by a
cleanup script, but it is anyone's guess which one.

Technically, this was not a sysadmin blunder, but it underscores the need
for individual accounts and for remote logging of ALL uses of group
accounts, including those performed by scripts.

It also underscores the absolute requirement that all scripts have error
trapping mechanisms. In this case, the rm -fr was likely preceded by a cd to
a nonexistent directory. Since this was performed as root, the script cd'd
to / instead. The rm -fr then removed everything. The other possibility is
that it applied itself to another directory, but, again, root privileges
allowed it to climb up the directory tree to root.

Creative uses of rm

See also Syaadmin Horror Stories

• From: [email protected] (Bill Broadley)
  Organization: University of Pittsburgh

  I was deleting last semesters users to try to dig up some disk space, I also deleted some test users at the same time.

  One user took longer then usual, so I hit control-c and tried ls. "ls: command not found"

  Turns out that the test user had / as the home directory and the remove user script just happily blew away the whole disk.

  ftp, telnet, rcp, rsh, etc were all gone. Had to go to tapes, and had one LONG rebuild of X11R5.

  Fortunately it wasn't our primary system...

• From: [email protected] (Chris Calabrese)
  Organization: AT&T Bell Labs, Murray Hill, NJ, USA

  We have a home-grown admin system that controls accounts on all of our machines. It has a remove user operation that removes the user from all machines at the same time in the middle of the night.

  Well, one night, the thing goes off and tries to remove a user with the home directory '/'. All the machines went down, with varying amount of stuff missing (depending on how soon the script, rm, find, and other importing things were clobbered).

  Nobody knew what what was going on! The systems were restored from backup, and things seemed to be going OK, until the next night when the remove-user script was fired off by cron again.

  This time, Corporate Security was called in, and the admin group's supervisor was called back from his vacation (I think there's something in there about a helicopter picking the guy up from a rafting trip in the Grand Canyon).

  By chance, somebody checked the cron scripts, and all was well for the next night...

• From: [email protected] (Tim Smith)
  Organization: University of Washington, Seattle

  I was working on a line printer spooler, which lived in /etc. I wanted to remove it, and so issued the command "rm /etc/lpspl." There was only one problem. Out of habit, I typed "passwd" after "/etc/" and removed the password file. Oops.

  I called up the person who handled backups, and he restored the password file.

  A couple of days later, I did it again! This time, after he restored it, he made a link, /etc/safe_from_tim.

  About a week later, I overwrote /etc/passwd, rather than removing it.

  After he restored it again, he installed a daemon that kept a copy of /etc/passwd, on another file system, and automatically restored it if it appeared to have been damaged.

  Fortunately, I finished my work on /etc/lpspl around this time, so we didn't have to see if I could find a way to wipe out a couple of filesystems...

• From: [email protected] ( bill pociengel)
  Organization: University of Minnesota

  After a real bad crash (tm) we got to test our backup by doing:

  # cd /
  # rm -rf *

  ohhhhhhhh sh*t i hope those tapes are good.

  Ya know it's kinda funny (in a perverse way) to watch the system just slowly go away.

• From: [email protected] (Barrie Spence)
  Organization: DataCAD Ltd, Hamilton, Scotland

  My mistake on Sun was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed "rm -rf ./.*".

  I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first.

• From: [email protected] (Contractor Bob Johnson)
  Organization: Tinker Air Force Base, Oklahoma

  Cleaning out an old directory, I did 'rm *', then noticed several files that began with dot (.profile, etc) still there. So, in a fit of obtuse brilliance, I typed...

  rm -rf .* &

  By the time I got it stopped, it had chewed through 3 filesystems which all had to be restored from tape (.* expands to ../*, and the -r makes it keep walking up the directory tree). Live and learn...

• From: [email protected] (John Rowe)
  Organization: Computer Unit. - University of Exeter. UK

  [email protected] (Rik Harris) writes: [snippet about "using 'find' in an auto-cleanup script which blew away half of the source" deleted. -ed.]

  If you're doing this using find always put -xdev in:

  find /tmp/ -xdev -fstype 4.2 -type f -atime +5 -exec rm {} \;

  This stops find from working its way down filesystems mounted under /tmp/. If you're using, say, perl you have to stat . and .. and see if they are mounted on the same device. The fstype 4.2 is pure paranoia.

  Needless to say, I once forgot to do this. All was well for some weeks until Convex's version of NQS decided to temporarily mount /mnt under /tmp... Interestingly, only two people noticed. Yes, the chief op. keeps good backups!

  Other triumphs: I created a list of a user's files that hadn't been accessed for three months and a perl script for him to delete them. Of course, it had to be tested, I mislaid a quote from a print statement... This did turn into a triumph, he only wanted a small fraction of them back so we saved 20 MB.

  I once deleted the only line from within an if..then statement in rc.local, the sun refused to come up, and it was surprisingly difficult to come up single user with a writeable file system.

  AIX is a whole system of nightmares strung together. If you stray outside of the sort of setup IBM implicitly assume you have (all IBM kit, no non IBM hosts on the network, etc.) you're liable to end up in deep doodoo.

  One thing I would like all vendors to do (I know one or two do) is to give root the option of logging in using another shell. Am I the only one to have mangled a root shell?

• From: [email protected] (Richard H. E. Eiger)
  Organization: Olivetti (Schweiz) AG, Branch Office Berne

  Just imagine having the sendmail.cf file in /etc. Now, I was working on the sendmail stuff and had come up with lots of sendmail.cf.xxx which I wanted to get rid of so I typed "rm -f sendmail.cf. *". At first I was surprised about how much time it took to remove some 10 files or so. Hitting the interrupt key, when I finally saw what had happened was way to late, though.

  Fortune has it that I'm a very lazy person. That's why I never bothered to just back up directories with data that changes often. Therefore I managed to restore /etc successfully before rebooting... :-) Happy end, after all. Of course I had lost the only well working version of my sendmail.cf...

• From: [email protected] (Gary Fowler)
  Organization: Evans & Sutherland Computer Corporation

  Once I was going to make a new file system using mkfs. The device I wanted to make it on was /dev/c0d1s8. The device name that I used, however, was /dev/c0d0s8 which held a very important application. I had always been a little annoyed by the 10 second wait that mkfs has before it actually makes the file system. I'm sure glad it waited that time though. I probably waited 9.9 seconds before I realized my mistake and hit that DEL key just in time. That was a near disaster avoided.
  [ I wish all systems were like that. Linux mkfs doesn't wait, but at ]
  [ least I have the source! -ed. ]

  Another time I wasn't so lucky. I was a very new SA, and I was trying to clean some junk out of a system. I was in /usr/bin when I noticed a sub directory that didn't belong there. A former SA had put it there. I did an ls on it and determined that it could be zapped. Forgetting that I was still in /usr/bin, I did an rm *. No 10 second idiot proofing with rm. Now if some one would only create an OS with a "Do what I mean, not what I say" feature.

  Gary "Experience is what allows you to recognize a mistake the second time you make it." Fowler

• From: [email protected] (Russell Street)
  Organization: University of Auckland, New Zealand.

  I once had "gnu-emacs" aliased to 'em' (and 'emacs' etc)

  One day I wanted to edit the start up file and mistyped
  # rm /etc/rc.local
  instead of the obvious.

  *Fortunately* I had just finished a backup and was now finding out the joys of tar and it's love of path names. [./etc/rc.local and /etc/rc.local and etc/rc.local) are *not* the same for tar and TK-50s take a *long* time search for non-existant files :(]

  Of course the BREAK (Ctrl-P) key on a VAX and an Ultrix manual and a certain /etc/ttys line are just a horror story waiting to happen! Especially when the VAX and manuals are in a unsupervised place :)

• From: [email protected] (Rik Harris)
  Organization: Monash University, Melb., Australia.

  Most of our disks reside on a single, high-powered server. We decided this probably wasn't too good an idea, and put a new disk on one of the workstations (particularly since the w/s has a faster transfer rate than the server does!). It's still really useful to be able to use all disks from the one machine, so I mounted the w/s disk on the server. I said to myself (being a Friday afternoon...see previous post) "it's only temporary.../mnt is already being used...I'll mount it in /tmp". So, I mounted on /tmp/a (or something). This was fine for a few hours, but then the auto-cleanup script kicked in, and blew away half of my source (the stuff over 2 weeks old). I didn't notice this for a few days, though. After I figured out what had happened, and restored the files (we _do_ have a good backup strategy), everything was OK.

  Until a few months later. We were trying to convince a sysadmin from another site that he shouldn't NFS export his disks rw,root to everyone, so I mounted the disk to put a few suid root programs in his home directory to convince him. Well, it's only a temporary mount, so....

  You guessed it, another Friday afternoon. I did a umount /tmp/b, and forgot about it. I noticed this one about halfway through the next day. (NFS over a couple of 64k links is pretty slow). The disk had not unmounted because it was busy...busy with two find scripts, happily checking for suid programs, and deleting anything over a week old. A df on the filesystem later showed about 12% full :-( Sorry Craig.

  Now, I create /mnt1, /mnt2, /mnt3.... :-)

  Remember....Friday afternoons are BAD news.

• From: [email protected] (Wm. L. Ranck)

  Well, after reading some of the stories in this thread I guess I can tell mine. I got an RS/6000 mod. 220 for my office about 6 months ago. The OS was preloaded so I had little chance to learn that process. Being used to a full-screen editor I was not happy with vi so I read in the manual that INED (IBM's editor for AIX) was full-screen and I logged in as root and installed it. I immediately started to play with the new editor and somehow found a series of keys that told the editor to delete the current directory. To this day I don't know what that sequence of keys was, but I was unfortunately in the /etc directory when I found it, and I got a prompt that said "do you want to remove this?" and I thought i was just removing the file I had been playing with but instead I removed /etc!

  I got the chance to learn how to install AIX from scratch. I did reinstall INED even though I was a little gun-shy but I made sure that whenever I used it from then on I was *not* root. I have since decided that EMACS may be a better choice.

• From: [email protected] (root)
  Organization: CRI, institute for telecommunication and computerservices.

  Well, waddya know... Some half hour ago, coming back from root (I was installing m4 on our system) [Shit, all my neato emacs tricks won't work. Damn, damn, damn kill, kill, KILL] to my own userid, I got this little message: "Can't find home directory /mnt0/crissl." and an other: "Can't lstat .". [Grrrrr, ^S and ^Q haven't been remapped...]

  Guess what happened, not an hour ago... A collegue of mine was emptying some directories of computer-course accounts. As I did a "ps -t" on his tty, what did I see? "rm -rf .*"

  Well, I'm not alone, he got sixteen other homedirectories as well. And guess what filesystems we don't make incremental backups of... And why not? Beats me...

  I haven't killed him yet, he first has to restore the lot.

  And for those "touch \-i" fans out there: you wouldn't have been protected...

• From: [email protected] (J. McPherson)
  Organization: Australian National University

  A few months ago in comp.sys.hp, someone posted about their repairs to an HP 7x0, after a new sysadmin had just started work. They {the new person} had been looking throught the file system to try to make some space, saw /dev and the mainly 0 length files therein. Next command was "rm -f /dev/*" and they wondered why they couldn't login ;)

  I think the result was that the new person was sent on a sysamin's course a.s.a.p

• From: [email protected] (Mark Brader)
  Organization: SoftQuad Inc., Toronto, Canada

  > ... if you're trying rm -rf / you'll NEVER get a clear disk - at least
  > /bin/rm (and if it reached /bin/rmdir before scanning some directories
  > then add a lot of empty directories). I've seen it once...

  Then it must be version-dependent. On this Sun, "cp /bin/rm foo" followed by "./foo foo" does not leave a foo behind, and strings shows that rm appears not to call rmdir (which makes sense, as it can just use unlink()).

  In any case, I'm reminded of the following article. This is a classic which, like the story of Mel, has been on the net several times; it was in this newsgroup in January. It was first posted in 1986.

  Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters....

  It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg. My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed
  grep 147 /etc/passwd
  only to receive the response
  /etc/passwd: No such file or directory.

  Instantly, I guessed that something was amiss. This was confirmed when in response to
  ls /etc
  I got
  ls: not found.
  I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager.

  When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines:
  # cd
  # rm -rf *

  Oh, shit, I thought. That would just about explain it.

  I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and typing

  cd /
  echo *

  I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched.

  Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck.

  Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable.

  So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack.

  By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working.

  Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference:

  070100002c000000000000000000000000000000000000000000000000000000
  0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
  8800040000bc012f65746300

  I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved.

  So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history.

  Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one.

  The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat.

  And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together.

  I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to:
  Mario Wolczko

• From: [email protected] (Stephen Samuel)
  Organization: University of British Columbia, Canada

  Some time ago, I was editing our cron file to remove core more than a day old. Unfortunately, thru recursing into VI sessions, I ended up saving an intermediate (wron) version of this file with an extra '-o' in it.

  find / -name core -o -atime +1 -exec /bin/rm {} \;

  The cute thing about this is that it leaves ALL core files intact, and removes any OTHER file that hasn't been accessed in the last 24 hours.

  Although the script ran at 4AM, I was the first person to notice this, in the early afternoon.. I started to get curious when I noticed that SOME man pages were missing, while others were. Up till then, I was pleased to see that we finally had some free disk space. Then I started to notice the pattern.

  Really unpleasant was the fact that no system backups had taken place all summer (and this was a research lab).

  The only saving grace is that most of the really active files had been accessed in the previous day (thank god I didn't do this on a saturday). I was also lucky that I'd used tar the previous day, as well.

  I still felt sick having to tell people in the lab what happened.

• From: Stephen Samuel ([email protected])
  Organization: University of British Columbia, Canada

  As some older sys admins may remember, BSD 4.1 used to display unprintable characters as a questionmark.

  An unfortunate friend of mine had managed to create an executable with a name consisting of a single DEL character, so it showed up as "?*".

  He tried to remove it.

  "rm ?*"
  he was quite frustrated by the time he asked me for help, because he had such a hard time getting his files restored. Every time he walked up to a sys-admin type and explained what happened, they'd go "you did WHAT?", he'd explain again, and they'd go into a state of uncontrolable giggles, and he'd walk away. I only giggled controlably.

  This was at a time (~star wars) when it was known to many as "the mythical rm star".

• From: [email protected] (J.J. Reynolds)
  Organization: Consolidated Traffic Management Services (CTMS)

  The SCO man page for the rm command states:

  It is also forbidden to remove the root directory of a given file system.

  Well, just to test it out, I one day decided to try "rm -r /" on one of our test machines. The man page is correct, but if you read carefully, it doesn't say anything about all of the files underneath that filesystem....

• From: [email protected] (Brooks Cutter)

  A while back I installed System V R4 on my 386 at home for development purposes... I was compiling programs both in my home directory, and in /usr/local/src ... so in order to reduce unnecessary disk space I decided to use cron to delete .o files that weren't accessed for over a day...

  I put the following command in the root cron...

  find / -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \;

  (instead of putting)

  find /home/bcutter -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \;
  find /usr/local/src -type f -name \*.o -atime +1 -exec /usr/bin/rm -f {} \;

  The result was that a short time later I was unable to compile software. What the first line was doing was zapping the files like /usr/lib/crt1.o .. and later I found out all the Kernel object files...

  OOPS! After this happened a second time (after re-installing the files from tape) I tracked down the offending line and fixed it....

  Yet another case of creating work by trying to avoid extra work (in this case a second find line).

There's an interesting thread on the CentOS mailing list on sys admin horror stories. Amusing read.

http://lists.centos.org/pipermail/centos/2010-June/095423.html

[Ilugc] [OT] sys admin horror stories

I once had a vendor sys admin run an "rm -rf ./project/ subdir" and delete the entire ./project folder and 2 months of live data.

Same vendor had also not noticed that automated backups had failed for many weeks and we were running without any backup for a long time.

A penalty of Rs 2 lakhs was enforced, all because of one extra space in the command.

- Raja

Greatest blunders

IT Resource Center

Christian Gebhardt

Hi
As a newby in UNIX I had an Oracle Testinstallion on a production system
productiv directory: /u01/...
test directory: /test/u01/...

deleting the test installation:
cd /test
rm /u01

OOPS ...

After several bdf commands I noticed that the wrong lvol shrinks and stops the delete command with Ctrl'C

The database still worked without the most binaries and libraries and after a restore from tape without stopping and starting the database all was ok.

I love oracle ;-)

Chris

harry d brown jr

Learning hpux? Naw, that's not it....maybe it was learning to spell aix?? sco?? osf?? Nope, none of those.

The biggest blunder:

One morning I came in at my usual time of 6am, and had an operator ask me what was wrong with one of our production servers (servicing 6 banks). Well nothing worked at the console (it was already logged in as root). Even a "cat *" produced nothing but another shell prompt. I stopped and restarted the machine and when it attempted to come back up it didn't have any OS to run. Major issue, but we got our backup tapes from that night and restored the machine back to normal. I was clueless (sort of like today)

The next morning, the same operator caught me again, and this time I was getting angry (imagine that). Same crap, different day. Nothing was on any disk. This of course was before we had raid availble (not that that would have helped). So we restored the system from that nights backups and by 8am the banks have their systems up.

So now I have to fix this issue, but where the hell to start? I knew that production batch processing was done by 9PM, and that the backups started right after that. The backups completed around 1am, which were good backups, because we never lost a single transaction. But around 6am the stuff hit the fan. So I had a time frame: 1am-6am, something was clobbering the system. I went though the crons, but nothing really stood out, so I had to really dive into them. This is the code (well almost) I found in the script:

cd /tmp/uniplex/trash/garbage
rm -rf *

As soon as I saw those two lines, I realized that I was the one that had caused the system to crap out every morning. See, I needed some disk space, and I was doing some house cleaning, and I deleted the sub-directory "garbage" from the /tmp/uniplex/trash" directory. Of course the script is run by root, which attempted to "CD" to a non-existent directory, which failed, and cron was still cd'd to "/", it then proceeded to "rm -rf *" my system!

live free or die
harry

Bill Hassell

I guess my blunder sets the record for "most clobbered machines" in one day:

I created an inventory script to be used in the Response Center to track all the systems across the United States (about 320 systems). These are all test and problem replication machines but necessary for the R/C engineers to replicate customer problems.

The script was written about 1992 to handle version 7.0 and higher. About 1995, I had a number of useful scripts that it seemed reasonable to drop these into all 300 machines as a part of the inventory process (so far, so good). Then about that time, 10.01 was released and I made a few changes to the script. One was to change the useful script location from /usr/local/bin to /usr/contrib/bin because of bad directory permissions. I considered 'fixing' the bad permissions but since these systems must represent the customer environment, I decided to move everything.

Enter the shell option -u. I did not use that option in my scripts and due to a spelling error, an environment variable was used in rm -r which was null, thus removing the entire /usr/local directory on 320 machines overnight.

Needless to say, I never write scripts without set -u at the top of the script.

Dave Johnson

Here is my worst.
We us BC's on our XP512. We stop the application, resync the BC, split the BC, start the application, mount the BC on same server, start backup to tape from BC. Well I had to add a LUN to the primary and BC. I recreated the BC. I forgot to change the script that mounts the BC to include the new LUN. The error message vgimport when you do not include all the LUN's is just a warning and it makes the volume group available. The backups seemed to be working just fine.
Well 2 months go by. I did not have enough available disk space to test my backups. (That has been changed). Then I decided to be proactive about deleted old files. So I wrote a script:
cd /the/directory/I/want/to/thin/out
find . -mtime +30 -exec rm {} \;

Well that was scheduled on cron to run just before backups one night. The next morning I get the call the system is not responding. (I guessed later the cd command had failed and the find ran from /).
After a reboot I find lots of files are missing from /etc /var /usr /stand and so on. No problem, just rebuild from the make_recovery tape created 2 nights before then restore the rest from backup.
Well step 1 was fine, but the backup tape was bad. The database was incomplete. It took 3 days (that is 24 hours per day) to find the most recient tape with a valid database. Then we had to reload all the data. After the 3rd day I was able to turn over recovery to the developers. It took about a week to get the application back on-line.
I have sent a request to HP to have the vgimport command changed so a vgimport that does not specify all the LUN's will fail unless some new command line param is used. They have not yet provided this "enhancement" as of the last time I checked a couple of months ago. I now test for this condition and send mail to root as well as fail the BC mount if it does.

Life lesson: TEST YOUR BACKUPS!!

RAC

Well I was very very new to HP-UX. Wanted to set up PPP connection with a password borrowed from a friend so that I could browse the net.

Did not worry that the remote support modem can not dial out from remote support port.
Went through all documents available, created device files dozen times, but never worked. In anguish did

rm -fr `ltr|tail -4|Awk '{print $9}'

(That to pacify myself that I know complex commands)

But alas, I was /sbin/rc3.d.

Thought this is not going to work and left that.
Other colleage not aware of this rebooted the system for Veritas netbackup problem.

Within next two hours HP engineer was on-site. Was called by colleague.

Was watching whole recovery process, repeatedly saying "I want to learn, I want to learn"

Then came to know that can not be done.

RD

Ramkumar Devanathan

I deleted /etc/* once. (just a few config files, eh?? nope, had to reinstall the OS);-)

Dario

Got a page once at 4:00AM about a filesystem almost full. Got up and started working on it without checking if someone else was in the system. I was recalling my commands using ESC-k but my supervisor was removing files in a different directory using rm -r *. The rest is history.

Gary Cantwell

A trainee student of one of my customers deleted /sbin on a HP-UX 11 system and copied it back in from a HP-UX 10.20 system and tried to deny it...

He's no longer a trainee student :-)

Gary

YLTan

got another one;

Trying to be helpful....I use a find . -name *.log -exec rm -f {} \; to rid all unused log files in apps temp and log dir. bulinux - How do I prevent accidental rm -rf - - Server Faultt I got a bit creative and put this in root cronjobs and guess what!! the next day, some very angry DBA come banging...cos it removed all Oracle database re-do log files.

Spend the rest of the weekend restoring database....:(

Recommended Links

Softpanorama Recommended

Top articles

Sites

• bash - Safe Directory Delete - Unix & Linux Stack Exchange
• Cleaning Up Your -tmp...The Safe Way Issue 18
• Is it safe to delete stuff in /tmp (SUSE 10.2)
• delete - Can I safely remove all the files in /tmp - Ask Ubuntu
• linux - Delete all of -var-log - Server Fault

Saferm implementations

• safe-rm

  Excerpt from the web site:

  Safe-rm is a safety tool intended to prevent the accidental deletion of important files by replacing /bin/rm with a wrapper, which checks the given arguments against a configurable blacklist of files and directories that should never be removed.

  Users who attempt to delete one of these protected files or directories will not be able to do so and will be shown a warning message instead:

  $ rm -rf /usr
  Skipping /usr

• How to install safe-rm on Debian Unstable (Sid) – Kreation Next – Support
• saferm Safely remove files, moving them to GNOME/KDE trash instead of deleting by Eemil Lagerspetz
• shell - Make `rm` move to trash - Unix & Linux Stack Exchange
• linux - How do I prevent accidental rm -rf - - Server Fault

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: February 19, 2020