May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Relax-and-Recover on RHEL

Version 2.1, Dec 14, 2020

News Disaster Recovery Recommended Links FIT USB flash drives Missing backup horror stories Bare metal recovery of Linux systems Cloning systems, disks and partitions
Unix tape archiver (Tar) Tar options for bare metal recovery Baseliners Data deduplication Gzip Unix cpio Dump and restore
dd Remote backups using dd Recovery of lost files using DD Mount a partition from dd disk image rsync Backing Up And Restoring Linux With SystemImager Perl Backup Scripts
Cron and Crontab commands HP Data Protector Dell DRAC and VFLASH Data deduplication Sysadmin Horror Stories Humor Etc


Relax-and-Recover a bare metal disaster recovery and system migration software. It is a complex, somewhat brittle because of this complexity, and not well documented (as usual for open source packages).

Stable releases are available as tgz files from the Sourceforge Downloads page. Each tar file contain the main script that should go into /usr/sbin, the set of directories in /usr/shar/rear as well as documentation and sample config file. There is also a make file for installation.

At the core, ReaR is a a set of scripts written in BASH. Scripts themselves are well written, demonstrate a good BASH style and as such represent ultimate documentation to the package, so they do compensate for the low quality of the documentation. As the package is designed for sysadmin it is natural to assume that they are cable to study and debug, if necessary, bash  scritps.  In addition to that, rear provides several debugging options (see below) that  provides additional insight into its workflow and allow to debug individual modules by simulation invocation environment for them. 

Version 1.17 which Red Hat 6 supplies is broken for G7 version of HP servers: you can backup, but you can't restore. Interestingly enough it does work for HP G6 servers. If you discover this situation one option is to try to upgrade to version 2.x. If version 1.17 is installed, the upgrade can be done by simply copying /usr/sbin/rear executable and /usr/sbin/share tree from tar of version 2.x. Please note, that version 2.x has three digit numeric prefixes for scripts, while scripts in version 1.x have a two digit prefix. If this hack helps and you can restore the test server, you can write a script or make file to propagate those updates to all relevant servers.

If version 1.17 is installed, the upgrade can be done by simply copying /usr/sbin/rear executable and /usr/sbin/share tree from tar of version 2.x.

The quality of console messages in ReaR is rather low. They do not include the name of script that generated them and they often miss fatal errors. You can't rely on them, but the quality of log written to /var/log/rear is good. It is better to watch ReaR log then console messages. The package has a useful debugging mode (see Learning rear internals and debugging ) which can and should be used to learn more about this system.

There are several YouTube videos about this package which helps to understand it (with two of them being quite informative): 

  1. First steps with Relax-and-Recover (ReaR) A very good presentation . Probably the best I found on YouTube. Published on Mar 7, 2017 included git installation and experimentation with Rear
  2. 2017 Relax-and-Recover Automated Testing
  3. Other that might be useful  but mostly flaff

In any bare metal recovery utilities the devil is in details and relax-and-restore managed to account for most important details right. That's why despite it fragility it is a valuable backup solution:

One huge advantage of ReaR is  that is uses gnu tar, representing just a complex wrapper over tar functionality

By default ReaR uses GNU tar for actual backup. The main advantage of using tar is that it is a well known by all sysadmins utility which is very reliable. Another advantage is that you do not need to learn something new, as "stack overflow" for sysadmins is a real problem (especially in RHEL7 which is way too complex for mere mortals): too many, too complex packages to maintain. And that greatly increases the confidence that your backup will be usable when the disaster strike -- you can use it even if rear breaks during recovery due to some bugs (happened to me). Just boot from RHEL ISO into rescue mode and you can perform restore.

Wrapper scripts included in ReaR are quite complex. You need to learn how to experiment with them as the documentation is dismal. The key problem with the packages like Relax and Recover is that they create there own "universe" with complex, non-obvious rules of submitting additional parameters (for example tar exclude list). This idiotism is so widespread that it can be called a new normal for open source. Just look at systemd -- no respect to their predecessors whatsoever. Some of those implicit rules are never documented and require experimentation to understand; most can be understood only looking at the code and running test cases.

The key problem with the packages like Relax and Recover is that they create there own "universe" with complex, non-obvious rules of submitting additional parameters (for example tar exclude list). This idiotism is so widespread that it can be called a new normal for open source. Just look at systemd -- no respect to their predecessors whatsoever.

Documentation is dismal. For any complex setup you need to experiment to make things right.

So outside simplistic usage (format/mkrescue/mkbackup troika with default /etc/rear/local.conf file) you are in Alice in Wonderland environment and often it is unclear what ReaR will do in particular case without experimentation. One trick that I have found useful it look at Rear tmp directory on the fly after rear started. You will see multiple files in it some of them should include the parameters you supplied.

Another common sense option is to look at transferred bytes indicator. It is does not look right -- for example the amount of transfered bytes is bigger that you the total of your local filesystems -- you can be  sure that some parameters you supplied (typically the list of excluded filesystems) are wrong.

Note that even simple operation as format of USB drive has gotchas -- you need to supply the device name, not the partition name to it (as you boot form the device not a partition ;-). And it should be unmounted.  More sophisticated script would accept both and  ask via prompt is this your real intention to create bootable partion on the device.

In other words Rear if far from AKM47 type simplicity and reliability of backup (while plain vanilla Tar is close ;-). And unless you are willing to look at the code (you should -- it is well written code; a rear example of a good style of Bash programming ;-) you might never fully understand the capabilities of this software (not that everybody need to; simplistic usage is OK for many).

Still despite all those warts ReaR can be used, as default configuration that create the rescue disk alone is worth some studing of this software. Moreover more complex configuration can be debugged using try and error process. As linux does not have any built-in bare metal recovery utility other then DD (but with it you face problem writing zeros on unused parts of filesystem before making backup with it in order to save space) you actually do not have too many alternatives ;-).

The other important thing is that because it is well written BASH scripts you can learn a lot of useful staff about BASH programming experimenting with it. If you are willing to look at the code, of course.

While "added complexity" often kills such packages, the issues in bare metal recovery are complex and at least as a "poor man" bare metal backup this package is a good candidate. It allows you to understand what is involved and how to solve certain problems inherent in bare metal backup. And what is important is that in simple cases it works as expected. So if you do not need to deviate from typical operations too much, for example, just adding the exclusion of some filesystems from the backup, you are OK. But you need to test whether it is able to restore the system (which, of course, also is not given).  but if you tarball is right, you can always restore it without rear scripts. That's huge advantage as stakes in such cases are very high.

In a simplified way Rear is equivalent to booting to the rescue mode on your RHEL distribution DVD, restoring MBR (which should be a part of tarball anyway, most sysadmins write it into /root with dd), and then using a tar file on a USB drive to restore the system. But the devil is always in details.

I would like to stret again importance of the fact that at the core of Relax-and-Recover is tar -- an already known by you utility. That means that the backup created by ReaR can be restored outside Relax and Recover framework. This is a very positive feature. Again, if restore fails (as it does with the version 1.17.x shipped with RHEL 6.9 on HP Intel servers like DL580 G7) you still can use tarball for recovery. In other words all this edifice of configuration files and settings which make rear very convenient (and very brittle) can be thrown out in case of need.

The key idea of Relax-and-Recover is that it is possible to construct bootable image with mini OS on the USB drive using files already available on the harddrive of installed Linux system. If you add to this bootable mini disk a tar archive of system directories and some scripts, you can restore the system from a USB drive. But implementation is wrong. The right approach would be to copy the OS from the system that is backuped (which just 6GB or less) as all the necessary drivers are present and this installation will definitely works. With the approach that Relax-and-Recover uses, the constructed bootable image sometimes work, and sometime (on HP servers) does not. In the latter case you need to boot to the recovery option in the RHEL ISO and you will be able to continue the restoration process. So bugs in the restore script and configuration is not the end.

By default rear uses /tmp filesystem for creation of the backup image. If it is not sufficient you need to specify alternative location for the image for rear to work

Still when it work, the recovery image represents a pretty capable mini-OS. You can SSH to it, you can add utilities missing on the fly via SCP fro other servers, you can chroot the restored image. When it does not work the fastest way is to create a second USB with mini OS (no GUI) installed from DVD and use it instead (two USB ports are not the problem on most servers, even HP ;-). If some USB ports need to be used for keyboard and mouse use a USB hub.

You can't relax until you test whether the restore mode works on your server or not. It can and does blue screen on some servers. You still have the tarball so this is just a nuisance, but it kills "relax" part ;-)

You can't relax until you test whether the restore mode works on your server or not. It can and does blue screen on some servers ;-)

When it works you have a bare metal recovery solution created using basic Linux components. Of course you can always install minimal OS on the USB drive and create tar manually. On systems where relax and recover blue screen on recovery this in the next option to try. But it requires shutting the server down and rebooting it from installation DVD or remotely mounted ISO to create the image. You can also do it on the fly by copying the OS manually and then installing grub and modifying grub menu to appropriate harddrive. But this requires some knowledge of grub.

Relax-and-Recover uses tar with the set of tar options and exclude files to created the tarball which later would be used to restore the system. If backup is not too large it can be put on permanently inserted FIT form factor USB drive. The current upper size for this low profile form factor is 256 GB (SanDisk 256GB Ultra Fit USB 3.1 ). The same upper limit has metal bar form factor (SanDisk Ultra Flair). Larger size "bar" form factor like PNY Pro Elite 512GB scale to 512GB, but that does not make much sense  as the SSD drive of the same capacity costs less.  That means that you can backup on 256GB flash drive filesystem with approximately 512GB of used space (assuming 50% compression). After that it is better to use USB 3.0 External hard drives such as Samsung T1 or Samsung T3, which are almost twice cheaper.

Relax-and-Recover produces a bootable image (rescue image) + tar file (backup). This image restore partitioning of the system, including LVM based partitioning. Once that is done it initiates a restore from backup. Restores to different size of harddrives and slightly different hardware are possible. Relax-and-Recover can therefore be used:

Currently Relax-and-Recover supports various boot media (incl. ISO, PXE, OBDR tape, USB or eSATA storage), a large variety of network protocols (sftp, ftp, http, nfs, rsync, cifs).

USB and NFS are probably the most important of them (low profile USBs like FIT have are generally limited to 256GB, so larger backups should go to NFS, as the most transparent way of storing backups.

It also can be used with pre-existing backup tools such as Bacula, HP DataProtector, IBM TSM, Symantec NetBackup, but that place it outside of its main advantages area -- simple and reliable bare-metal backup solution. In this combination it is neither simple, nor reliable ;-)

Relax-and-Recover recovery process is brittle. The setup from RPM requires installation of additional RPMs.


Installation of RHEL7 package is pretty straightforward. There are some nuances with the installation  of rear of CentOS and RHEL6

  1. Please not that in CentOs7 you need explicitly specify installation of  syslinux-extlinux. (that might be the case of Debian too Package dependency error when backing up Debian 8.2 system to USB device · Issue #720 · rear-rear · GitHub )  see For some reason is is not installed as a dependency:
    yum install syslinux-extlinux rear
  2. Installation of RHEL6 is more involved (see below)

Installation on RHEL7

Here is protocol of installation for RHEL7 (6 dependent packages are installed):

[0]d620@ROOT:~ # yum install rear
... ... ... 

 Package                Arch              Version                      Repository       Size
 rear                   x86_64            2.4-2.el7                    base            636 k
Installing for dependencies:
 attr                   x86_64            2.4.46-13.el7                base             66 k
 bc                     x86_64            1.06.95-13.el7               base            115 k
 genisoimage            x86_64            1.1.11-25.el7                base            299 k
 libusal                x86_64            1.1.11-25.el7                base            136 k
 mtools                 x86_64            4.0.18-5.el7                 base            203 k
 syslinux               x86_64            4.05-15.el7                  base            990 k

Transaction Summary
Install  1 Package (+6 Dependent packages)

Total download size: 2.4 M
Installed size: 6.5 M
Is this ok [y/d/N]: y
... ... ... 

Installation on RHEL6 is not well documented

I spend several hours trying to install in of RHEL 6.8 until I figured out what to do. This is not how decent open source software should be documented and developers should be ashamed.  The installation on RHEL fails if the supplementary software channel is not enabled before you attempt the installation. But it fails in its own particular way. Software is installed successfully, you can format USB drive, but when you try to make backup you get them message that syslinux-extlinux package is missing.  Also it you download those packages (for batch installation on many servers) and try to install them using yum for some reason on RHEL 6.x you face incompatibility of libraries problem and installation fails. Among packages you need

On RHEL 6.5-6.8 you can install rear using the commands

yum-config-manager --enable repository rhel-6-server-optional-rpms
yum install rear syslinux-extlinux genisoimage syslinux

The process of recovery

The most typical and the simplest setup for recovery with ReaR is when your backup is on bootable USB drive or on rather small ISO file (most servers can boot from ISO file). You backup generally consist of two parts

To recover from a failed drive just boot from USB.   If you use NDF backup you need to boot from this ISO on NFS filesystem. If needed this ISO can be transferred to your PC and mounted as bootable image via ILO or DRAC.

You will be presented with the inital menu, click of the first (recover server name option). You will be presented with submenu that contain two recover option (many in several variants corresponding to different dates):

Manual gives you far more flexibility. The process is described in some details in the instructions provided Red Hat for RHEL and it will work on CentOS and other clones. Please note that RHEL the recovery is complicated by existence of SE Linux, if it is enabled.

Choosing manual recovery option you get into to the command prompt of a pretty capable mini-OS

Networking is enabled and SSH daemon is running, so you can SSH to it, which is huge advantage if you are working with old ILO or DRAC with split cursor.

Choosing manual recovery option you get into to the command prompt of a pretty capable mini-OS. Networking is enabled and SSH daemon is running, so you can SSH to it, which is huge advantage if you are working with old ILO or DRAC with split cursor.

As you have access to your network you can add any utilities you want but which are missing on the fly via SCP.

You can also chroot the the harddrive with root filesystem and /boot filesystem. In the latter case you can edit grub settings.  You can also reinstall grub (not sure how it works, if it works at all if /boot is a separate filesystem).

When the recovery does not work probably because mini-OS created does not boot correctly, the fastest workaround to create a second USB with mini OS (no GUI) installed from DVD and use it instead (two USB ports are not the problem on most servers, even HP). If you have insufficient USB ports or no USB ports ( on HP blades you need some stupid dongle) you can always use your PC to load the initial ISO from DRAC or ILO. If you have just one spare USB port you can use a USB hub.

Tar archive with data can be located on USB disk, of better on NFS filesystem. It can moved to hard drive and then unpacked (if you have enough disk space). Or some files can be extracted from it  for any purpose you wish, including recovery of some lost files.

Tar archive with data located on USB disk or, better, on NFS filesystem. It can moved to hard drive and then unpacked (if you have enough disk space). Or some files can be extracted from it for any purpose you wish, including recovery of some lost files.

There are some interesting possibilities to use Relax-and-Recover for migrating systems to slightly different hardware (which is often the case if computer is damaged beyond repair, for example some chip on motherboard failed. In this case Relax-and-Recover provides step-by-step menus which help make decisions to restore the OS and data in new server.

As source code is available you can look into it and experiment with running it via bash with -x option. It also has logging and optionally "extended output" mode which helps to understand the concepts behind Relax-and-Recover, troubleshoot during initial configuration and help debug during integration.


Quick Start Guide

In almost all circumstances you have to configure two main settings and their parameters: The BACKUP method and the OUTPUT method.

The backup method defines, how your data was saved and whether Relax-and-Recover should backup your data as part of the mkrescue process or whether you use an external application, e.g. backup software to archive your data.

The output method defines how the rescue system is written to disk and how you plan to boot the failed computer from the rescue system.

See /usr/share/rear/conf/default.conf for an overview of the possible methods and their options. For example


ReaR uses the following four variables to define how it backup particular system:

For NETFS method you have to define at least three variables for backup to be successful

Useful additional variables depends of your BACKUP_URL. We will discuss then for each of three BACKUP_URLs we cover in this page. One general useful setting is:

NETFS_KEEP_OLD_BACKUP_COPY=y  # If you want to keep the previous backup archive

Another is excluded filesystems. We will discuss it later (see Excluding large filesystems from the backup )

Variant 1: Backup to USB (creation of bootable recovery disk)

This is the most popular initial configuration of ReaR. You should proceed as following:

  1. Prepare your USB media. We assume that USB media is at /dev/sdb.

    Before formatting the device should be unmounted. The current version of rear does not unmount it and simply fails with a cryptic message

    Again, you better check this. The device should unmounted before you can start (USB disk are typically mounted on insertion to /media folder)

    Like always the case with format existing data will be destroyed. Formatting is very slow and can take for 32GB flash drive ten minutes on USB 2.0 connection. The drive is formatted as Ext3 filesystem.

    1. Here you need device name, not the partition name /dev/sdb1
    2. You can format the drive yourself without using REAR. That's probably the "best practice" with the current version.

    After you type Enter, Relax-and-Recover asks you to confirm that you want to format the device. If everything is correct and you do not accidentally specify you primary harddrive :-), then answer


    The most common reason for this command to abort with error is that the device is still mounted.

    If operation is successful, the device will be labeled REAR-000 by the format workflow. This is the default label and you change it via configuration file if you object to it.

  2. Edit the /etc/rear/local.conf configuration file. For example you can write initial set of pretty generic parameters using cat:
    cat > /etc/rear/local.conf <<EOF
    ### write the rescue initramfs to USB and update the USB bootloader
    ### create a backup using the internal NETFS method, using 'tar'
    ### write both rescue image and backup to the device labeled REAR-000

    If you have multiple boxes you can distribute this file to other boxes: the content of this file is server independent. If you already have such a file on other server, you can scp it to the new server you are configuring.

    Typically you also can benefit from the exclusion of certain directories, but not at this stage of learning.

  3. Make a backup of your /boot directory (see below). Normally sysadmin usually have such a backup, but on some servers they might nor. If not just do it (see below)
  4. CHECK that your USB drive is mountable and is not READ ONLY. That sometimes happens with USB drives that they switch to read-only mode and in this case Rear might happily delete /boot directory on your system.

    You need to verify that your USB drive used for backup can be mounted and is writable.
    That's important as Rear misbehaves if USB drive can't be mounted

    View the log immediately after the operation you specified ended and browse it for error messages

  5. Create a rescue image. This is essentially a test run as this is pretty quick operation. The size of rescue image is about 100MB. We want verbose output (-v option).
    sudo /usr/sbin/rear -v mkrescue
    You will get output similar to the following:
    Relax-and-Recover <version>
    Using log file: /var/log/rear/rear-<hostname>.log
    Creating disk layout
    Creating root filesystem layout
    Copying files and directories
    Copying binaries and libraries
    Copying kernel modules
    Creating initramfs
    Writing MBR to /dev/sdb
    Copying resulting files to usb location

    If you can create rescue image that means your rear installation works at least in this part, but you need to restore it at least once to be sure that respore part is OK too.

    NOTE: if you forgot to create configuration file rear will write rescue ISO image to /var/lib/rear/output/rear-`hostname`.iso

  6. Check the log file for possible errors. This is a VERY important step and should not be skipped -- rear is a brittle software.
    And it can play jokes with you.

    Checking the log is a very important step and should not be skipped -- rear is a brittle software.

    And it can play jokes with you when you need to recover your server

    I would also recommend to use some "we do not care about server" and try to restore it using your version of ReaR to make sure that it does not contain catastrophic bugs. Because you can encounter one at the most inopportune moment. But the structure of rear backup is useful even if restore for a particular version is not working. And the backup tar archive is usable too.

    That means that you can restore the system step by step yourself following the scripts rear provides. All the necessary information is still present that that&#rire39; the most important part of the whole exercise.

  7. Reboot your system and try to boot from the USB device. You should get a self-explaining boot menu. Verify that the server can boot itself after 30 sec delay. On HP 580 G7 with rear 1.17.2 I encountered a bug due to which it blow up with the message invalid opcode of something like that. You can try restore the OS if this is a test computer to ensure that the version of Relax and Recover you use does not contain some glaring errors.
  8. If your USB device has enough space, initiate a backup. Backup of filesystem containing, say 32GB on the flash drive is a long operation which take an hour or so so you need to submit it either in screen session (I experienced problems with running it with nohup prefix; also you will not see any output until it ends which is a bad idea) You can use the following command:
    /usr/sbin/rear -v mkbackup

As tar is gziped you get (depending on the content on you filesystems) approximately 2/3 smaller image the the size of filesystem. That means that 64GB flash can store backup of ~100GB data. With 256GB FIT drive (the max size of thise type of derive as of December 2017) you probably can backup half a terabyte of data.

If you have more then that you need to use iether NFS or rsync for storing backup on remote server.

You can also use USB drive instead of flash drive. 6TB Seagate USB 3.0 SMR drive (so it's supposed to be slow and not to be used as system drive - for archive purposes only) is now about $130 (at Amazon); A more traditional drive will cost you $180; one terabyte SSD drive is around $300.

One of the positive things about Relax and Restore is that the rescue image configures internet for you, so all shares that are available on "normal" server are accessible from the rescue image boot-up. It is not that difficult to configure network manually, but when it is done automatically for the you it is even better.

After you have such an image you are now better better prepared for catastrophic failure of harddrives or other server components. But if server has large partitions you need to use NFS of rsync to put the image on a remote filesystem. that would be the next step in learning rear.

Variant 2: Backup to a local filesystem

You can put your backup on any locally mounted filesystem. Similar to previous, but you define in configuration OUTPUT=ISO and directory to store your backup in BACKUP_URL (ATTENTION: this is a directory, not a file)


NOTE: Within this target directory rear will create subdirectory with the name of the server. So ISO will be in this subdirectory, not in the specified directory.

Variant 3: Backup using NFS

Of course, you can backup to any mounted NFS share using file method (see above), but in this case your backup will not be available during the restore operation as this filesystem will not be mounted. So in such cases you need to define BACKUP_URL (ATTENTION: this is a directory, not a file) in the form that can be used during the restore operation and provide IP parameters to endure that this server is assessable.
For example:

And since typically all servers use NTP for time synchronization, you should probably also add these lines to iether /etc/rear/local.conf or to /etc/rear/site.conf as this setting is typically site wise:

During restore operation, you need to have access to correct network settings. If you have DHCP on this segment you can use

Otherwise you need to ask Rear to save your static networking settings via the directive:


Example: NFS Share (

# Create ReaR rescue media as ISO image
# optionally define backup software, e.g. TSM, NBU, DP, BACULA
# the following is required on older VMware VMs
# MODULES_LOAD=( vmxnet )
# line below was automatically added by
# line below was automatically added by
# Enable the use of cfg2html (set to non-empty to disable). cfg2html must be
installed independently from ReaR.
# Warn about NETFS not being a professional backup solution
# additional tools
PROGS=( "${PROGS[@]}" lsof parted fdisk cfdisk )

Always check ReaR logs after the backup

You need always check your logs. The fact that you have no warnings on console does not means that your backup is successful. And if you do not read logs you might discover this sad fact only when you try to revocer failed system from backup.

Always check your logs. The fact that you have not warnings on console does not means that your backup is successful

During backup to USB drive the most typical error is the situation when there backup is lager then free space on the device left. If flash drive is used, most often the size of the backup simply exceeds the size of the USB drive used. If ReaR is aborted, it does not clean the space on the USB drive, so you need to do it manually or subsequent backup will fail.

What exactly happened often in not clear and only analysis of the log file can tell you exact situation and point to the root case.

So the analysis of the log file is a must.

Creating your local and site configuration: local.conf vs site.conf

To configure Relax-and-Recover you have to edit two configuration files in /etc/rear/

Settings that are common to all your server in particular segment of datacenter should generally go to site.conf (which is intended to be distributed to all servers in your backup scheme)

Unique for a particular server settings should go to local.conf (which is intended to capture unique settings which are necessary for some servers and which should not be propagated to other servers) .

Settings in local.conf overwrite settings in site.conf.

All other configuration files hold defaults for various distributions and should not be changed.

In /usr/share/rear/conf/ there are also some configuration files which you can study. They have extension .conf

The most useful for study is default.conf (which is also the largest). It is well commented and contains ton of variable that you can set (and hope that they work and you understand their semantics correctly, which is not given ;-). This "preoccupation with variables" is a weak point of ReaR, but we have what we have.

Using the site.conf does not makes any sense if you do not distribute it to all your systems. If you have identical configuration on all servers you can also distribute local.conf t o all servers instead.

In reality you are better off creating subdirectories for each of your rear setup and specifying this directory via option -c. For example you can create directory


And use it for backup to USB

And you can create directory


and use this directory for creating ISO on NFS

Excluding large filesystems from the backup

Often you need to exclude some filesystems from backup (typically large data filesystems that are backuped separately via rsync, or some other method)

By default ReaR excludes NFS mounted partitions from backup

The most useful for study is /usr/share/rear/conf/default.conf

Source view -etc-rear-default.conf

# Custom command backup stuff
# examples for external backup. In this mode your external program must do EVERYTHING
# In the example below we backup / to the vms host via tar and netcat
# NOTE: The EXTERNAL_* commands can be also defined as an array () to better protect
# arguments with blanks
# NOTE: The EXTERNAL_* commands will be run inside eval like this:
# eval "${EXTERNAL_BACKUP[@]}"
# Command to backup the required data
# This example saves the data via SSH to a remote system called vms
EXTERNAL_BACKUP="tar -c -l -z / | ssh vms 'cat >rear64/backup.tar.gz'"
# Command to restore the data
EXTERNAL_RESTORE="ssh vms cat rear64/backup.tar.gz | tar -C /mnt/local -x -z"
# Command to verify the availability of the backup resource, will be executed only if PING=1
# NOTE: This command will be run within the ProgressBar system !! You should therefore avoid
# output on STDOUT by rerouting that to FD 8, the progress bar
EXTERNAL_CHECK="ssh vms date 1>&8"
# How to exclude something ----- EXCLUDES -------
# You cannot exclude a device (e.g. /dev/sdg) directly. Instead you have to exclude everything
# ON that device and then the dependancy tracker will automaticall exclude the device from the
# recovery (because there won't be any recovery information for that "unnecessary" device).
# Furthermore, you have to exclude MD devices and LVM2 volume groups separately as there is no
# automatic detection of these dependancies (yet, please write and submit it !)
# Exclude filesystems by specifying their mountpoints. Will be automatically added to the
# $BACKUP_PROG_EXCLUDE array during backup to prevent the excluded filesystems' data to
# be backed up
# examples: /tmp
#           /media/bigdisk
# Exclude MD devices
# examples: /dev/md0
#           /dev/md/0
# Exclude LVM2 volume groups. This will automatically exclude also the creation of the corresponding
# physical and logical volumes that belong to the excluded volume group.
#       Y O U   H A V E   B E E N   W A R N E D  ! ! !

You can use option EXCLUDE_MOUNTPOINTS to exclude some mounted filesystems. For example:

EXCLUDE_MOUNTPOINTS=( '/Usbdrive' '/Apps' '/Scratch' ) 

This is a BASH array which supposedly is added to "main exclusion array" which is BACKUP_PEXCLUDE (and which already contains several components such as /tmp, so you should not overwrite it)


You could also try running rear with the -d switch. In your /tmp/rear-XXX/ folder should be a backup-exclude.txt file. It's in tar exclude syntax. That's an interesting file to study, because if it is incorrect your tar backup will fail miserably.  Using rear dump is not that helpful as it does not include those parameters

To illustrate a mess we are dealing with look at the discussion that I have found Excluding a folder in a rear backup · Issue #216 · rear-rear · GitHub

cocampbe commented on Mar 27 2013

I've been perusing the docs and have not come up with a solution yet. I am trying to exclude a folder that contains a large amount of logs. Here is what I have in the local.conf.
ONLY_INCLUDE_VG=( VolGroup00 )
EXCLUDE_COMPONENTS=( "fs:/usr/openv/logs/nbftsrvr" )

So far this does not work. I am not sure if the order matters, or if I am using the wrong exclude directive. Any insight is greatly appreciated.

 jhoekx on Mar 27 2013

EXCLUDE_COMPONENTS only works for real system components like filesystems or logical volumes. If you just want to exclude a directory you have to use the BACKUP_PROG_EXCLUDE array.

cocampbe on Mar 27 2013

Cool. I will do that. I was about to try that, but I just wasn't sure. I also thought about just trying EXCLUDE_MOUNTPOINTS even though it technically is not.

jhoekx on Mar 27 2013

Can you show us the local.conf you're using? To make sure the BACKUP_PROG_EXCLUDE syntax is OK.

cocampbe on Mar 27 2013

ONLY_INCLUDE_VG=( VolGroup00 )
BACKUP_PROG_EXCLUDE=( /usr/openv/logs/nbftsrvr )

cocampbe on Mar 27 2013

I just added single quotes to BACKUP_PROG_EXCLUDE=( '/usr/openv/logs/nbftsrvr' ). trying again.

jhoekx on Mar 27 2013

The line is not entirely correct. It should be:

BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/usr/openv/logs/nbftsrvr' )

You could also try running rear with the -d  switch. In you /tmp/rear-XXX/ folder should be a backup-exclude.txt file. It's in tar exclude syntax. That's an interesting file to share.

cocampbe on Mar 27 2013

I checked the file and it contains /usr/openv/logs/nbftsrvr.

jhoekx on Mar 27 2013

The file is sent to tar as-is. Maybe try /usr/openv/logs/nbftsrvr/*?

cocampbe on Mar 27 2013

File is now showing /usr/openv/logs/nbftsrvr/*. I am running rear again.

cocampbe on Mar 27 2013

I just tailed the backup.log file and saw that it is not copying those files. So it is working. Looks like I may be backing up something else that is making the archive rather large. Thanks for your help.

cocampbe on Mar 27 2013

Yep. even though I have ONLY_INCLUDE_VG=( VolGroup00 ), it was backing up a san attached volume. I added it to the exclude list.

rear-06-layout-configuration.adoc at master · rear-rear · GitHub

Manual excludes

It seems prudent to prevent the external drives from ever being backed-up or overwritten. The default configuration contains these lines:
# Exclude components from being backed up, recreation information is active

# Exclude components during component recreation
# will be added to EXCLUDE_BACKUP (it is not backed up)
# recreation information gathered, but commented out

# Exclude components during the backup restore phase
# only used to exclude files from the restore.
To prevent an inadvertently mounted backup filesystem being added to the restore list, the easiest way is to add the filesystem to the EXCLUDE_RECREATE array.
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/media/backup" )

Overcomplexity: the road to hell is paved with good intentions

While the motivation of the creators of this backup software were noble, the net result is  questionable because complexity of the resulting scripts bite them in the back. There was not real effort to ensure transparency of the interaction and parameter passing between the scripts and tar.

The architecture chosen (complex scripts driven by million of env variables) is questionable and ability to correct errors in generated commands (for example in the tar commend used for the backup) is limited. There are also complex interdependencies, and not all situations are diagnosed on the console

In pretty common case when that tarball overflow USB drive, there no explicit diagnostics about this important event on the console and if you do not check logs you can be royally screwed.

When you exceed certain level of complexity with bash scripts the resulting "mess" is not only unmanageable, it is outright dangerous. Unless you are happy with the default "mkrestore" option, you can be royally screwed (especially if you fail to look at the logs). Backup can fail due to the lack of space on, say, USB drive, but there will be no diagnostic messages. Or backup can succeed, but the content of the tarball will be not what you expect. As the result, if you do not check that content of the tarball beforehand, you can destroy some filesystems during restore operation by restoring filesystems that you did not intent to restore. There are several other horror stories that can happen, when the complexity of the scripts far exceed you level of understanding and generated command are not explicitly listed.

Here is one example. In most real cases you need to exclude some filesystems and tar provides this capability. But with the maze of scripts in ReaR this capability is not available directly in ReaR. You need to populate a special array EXCLUDE_MOUNTPOINTS :

EXCLUDE_MOUNTPOINTS=( '/Usbdrive' '/Apps' '/Scratch' ) 

You can't provide you own exclude file, you can't correct generated command with you own script, unless you want to mess with the source code. And you need to rely on variable provided by ReaR, which depends on the processing logic of corresponding script so without looking at the script code you will be trying to find a black can in dark room. The logic in the script that govern processing of this array

# backup all local filesystems, as defined in mountpoint_device

for k in "${BACKUP_PROG_INCLUDE[@]}" ; do
	test "$k" && echo "$k"
done > $TMP_DIR/backup-include.txt
# add the mountpoints that will be recovered to the backup include list
while read mountpoint device junk ; do
	if ! IsInArray "$mountpoint" "${EXCLUDE_MOUNTPOINTS[@]}" ; then
		echo "$mountpoint"
done <"$VAR_DIR/recovery/mountpoint_device" >> $TMP_DIR/backup-include.txt

# exclude list
for k in "${BACKUP_PROG_EXCLUDE[@]}" ; do
	test "$k" && echo "$k"
done > $TMP_DIR/backup-exclude.txt
# add also the excluded mount points to the backup exclude list
for k in "${EXCLUDE_MOUNTPOINTS[@]}" ; do
	test "$k" && echo "$k/"
done >> $TMP_DIR/backup-exclude.txt

Note that the script adds a trailing backlash to each excluded mount point at line 19.  So  you should not include trailing backslash in your spec.

Generally this idea of using approximately a hundred variables (yes around a hundred) that guide the behaviour of ReaR sucks. To add the insult to injury this capability is not well documented and the logic is too simplistic and requires some additional work

If you put additional slash at the end, or make some other minor mistake the configuration will not work, but no errors will be generated.

the way to check the result is to run manually your exclude file with tar. The backup-exclude.txt file in can be found in  tmp directory used by ReaR and if it is not right you can adjust you input. Here is the list of file in this directory (/tmp/rear.8T1ZsZOFSXWwA6g/tmp/ in our case)

-rwxrwx---+ 1 nnb None 35646232 Apr  3 10:08 backup.log
-rwxrwx---+ 1 nnb None       41 Apr  3 05:37 backup-exclude.txt
-rwxrwx---+ 1 nnb None       18 Apr  3 05:37 backup-include.txt
drwxrwx---+ 1 nnb None        0 Apr  3 16:51 boot
-rwxrwx---+ 1 nnb None       43 Apr  3 05:36 bootloader
-rwxrwx---+ 1 nnb None       28 Apr  3 05:36 copy-as-is-exclude
-rwxrwx---+ 1 nnb None   179899 Apr  3 05:36 copy-as-is-filelist
-rwxrwx---+ 1 nnb None 82642591 Apr  3 05:37 initrd.cgz
drwxrwx---+ 1 nnb None        0 Apr  3 16:51 mappings
-rwxrwx---+ 1 nnb None       93 Apr  3 05:36 parted
-rwxrwx---+ 1 nnb None       31 Apr  3 05:36 partitions
-rwxrwx---+ 1 nnb None       18 Apr  3 05:36 partitions_unsorted
-rwxrwx---+ 1 nnb None       18 Apr  3 05:36 partitions-data

The default content is as following


Unlike the default content, all generated entries does not have * at the end, only a slash is added, for example


If you did something wrong it will be visible in log stored in temp directory (/tmp/rear.8T1ZsZOFSXWwA6g/tmp/ in our case) in the structure of tar command and in this case you need to cancel the backup and change the configuration so that it works as intended

here is an example of botched tar command which included "undesirable" /Usbdrive filesystem into the backup:

2018-04-03 05:37:48.851184722

tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash \ 
  --one-file-system --ignore-failed-read --anchored --gzip -X /tmp/rear.8T1ZsZOFSXWwA6g/tmp/backup-exclude.txt \
  -C / -c -f - /home / /Usbdrive /var/log/rear/rear-z17.log \
  | cat BACKUP_PROG_CRYPT_KEY | dd of=/tmp/rear.8T1ZsZOFSXWwA6g/outputfs/rear/z17/20180403.0536/backup.tar.gz

Tar option -C in tar means change to directory DIR, so this command is running from the root directory. You can generally test that command separately (in this case you do not need to pipe it to dd of course)

At this point your enthusiasm about this package is probably somewhat deflated. But I would like to stress that mkrecover option is very useful and works. And you get the tar command that failed, modify it and rerun getting the tarball you need. That's the advantage of using tar, although architectural decisions taken in ReaR are simplistic (usage of huge number of env variable for driving the scripts) are definitely wrong and the package is suffering from overcomplexity.

For more complex cases you do need to know bash well to be able to debut this beast.

Another problem is that ReaR does not allow just generate commands it uses, which would allow correction of some subtle errors by your own scripts. But the fact that you can extract the generated tar command from the log is hugely positive and greatly simplifies the debugging.

You can also run you own tar command based on the example you have found in the log instead of mkbackup and incorporate such a command in your own script

This is a BASH array which supposedly is added to to "main exclusion array" which is BACKUP_PROG_EXCLUDE (and which already contains sevral components such as /tmp, so you should not overwrite it)

Rear cron job which is installed by RHEL package

Rear has a cron job installed when RHEL package is installed. It adds the file rear to the /etc/cron.d. directory

The idea is to update rescue image automatically each time the layout of the box has been changed (for example you updated the version of OS) which is a good idea:

30 1 * * * root /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue

Rear layout in RHEL

ReaR RPM is more or less LSB complaint and puts files in expected places:

ReaR uses several locations during backup process ( ):

Learning rear internals and debugging

There is two presentations that help you to understand how to experiment with ReaR:

The simplest way is to create two virtual machines and backup on one and recover on the another.

There are several options that help you with the debugging:

To view/verify your configuration, run
rear dump

It will print out the current settings for BACKUP and OUTPUT methods and some system information.

Relax-and-Recover is built as a modular framework. A call of rear <command> will invoke the following general workflow:

  1. Configuration: Collect system information to assemble a correct configuration (default, arch, OS, OS_ARCH, OS_VER, site, local). See the output of rear dump for an example. + Read config files for the combination of system attributes. Always read 'default.conf' first and 'site.conf', 'local.conf' last.
  2. Create work area in '/tmp/rear.$$/' and start logging to '/var/log/rear/rear-hostname.log'
  3. Run the workflow script for the specified command: '/usr/share/rear/lib/<command>'
  4. Cleanup work area

The result of the analysis is written into configuration files under '/etc/rear/recovery/'. This directory is copied together with the other Relax-and-Recover directories onto the rescue system where the same framework runs a different workflow - the recovery workflow.

The recovery workflow is triggered by the fact that the root filesystem is mounted in a ram disk or tmpfs. Alternatively a "demo" recovery workflow can be started manually. This will simply recover all data into a subdirectory and not touch the hard disks (Phase 2).

The recovery workflow consists of these parts (identically named modules are indeed the same):

  1. Config: By utilizing the same configuration module, the same configuration variable are available for the recovery, too. This makes writing pairs of backup/restore modules much easier.
  2. Verify: Verify the integrity and sanity of the recovery data and check the hardware found to determine, whether a recovery will be likely to succeed. If not, then we abort the workflow so as not to touch the hard disks if we don’t believe that we would manage to successfully recover the system on this hardware.
  3. Recreate: Recreate the FS layout (partitioning, LVM, raid, filesystems, …​) and mount it under /mnt/local
  4. Restore: Restore files and directories from the backup to '/mnt/local/'. This module is the analog to the Backup module
  5. Finalize: Install boot loader, finalize system, dump recovery log onto '/var/log/rear/' in the recovered system.

FS layout

Relax-and-Recover tries to be as much LSB compliant as possible. Therefore ReaR will be installed into the usual locations:

Main program
Internal scripts
Build area

Layout of /etc/rear

Default configuration - will define EVERY variable with a sane default setting. Serves also as a reference for the available variables 'site.conf' site wide configuration (optional)
local machine configuration (optional)
$(uname -s)-$(uname -i).conf
architecture specific configuration (optional)
$(uname -o).conf
OS system (e.g. GNU/Linux.conf) (optional)
OS and OS Version specific configuration (optional)
Directory to keep user-changeable templates for various files used or generated
template for pxelinux.cfg per-node configurations
isolinux.cfg template
other templates as the need arises
Recovery information

Layout of /usr/share/rear

default rescue FS skeleton
skel/$(uname -i)/
arch specific rescue FS skeleton (optional)
OS-specific rescue FS skeleton (optional)
Backup-SW specific rescue FS skeleton (optional)
Output-Method specific rescue FS skeleton (optional)
function definitions, split into files by their topic
prep/$(uname -i)/*.sh
Prep scripts. The scripts get merged from the applicable directories and executed in their alphabetical order. Naming conventions are: + + where 00 < < 99
Scripts to compare the saved layout (under /var/lib/rear/layout/) with the actual situation. This is used by workflow rear checklayout and may trigger a new run of rear mkrescue or rear mkbackup
Scripts to capture the disk layout and write it into /var/lib/rear/layout/ directory
Analyse-Rescue scripts: …​
Build scripts: …​
Pack scripts: …​
Backup scripts: …​
Output scripts: …​
Verify the recovery data against the hardware found, whether we can successfully recover the system
Recreate file systems and their dependancies
Restore data from backup media
Finalization scripts

Inter-module communication

The various stages and modules communicate via standardized environment variables:

NAME TYPE Descriptions Example
CONFIG_DIR STRING (RO) Configuration dir '/etc/rear/'
SHARE_DIR STRING (RO) Shared data dir '/usr/share/rear/'
BUILD_DIR STRING (RO) Build directory '/tmp/rear.$$/'
ROOTFS_DIR STRING (RO) Root FS directory for rescue system '/tmp/rear.$$/initrd/'
TARGET_FS_ROOT STRING (RO) Directory for restore '/mnt/local'
PROGS LIST Program files to copy bash ip route grep ls …​
MODULES LIST Modules to copy af_unix e1000 ide-cd …​
COPY_AS_IS LIST Files (with path) to copy as-is '/etc/localtime' …​

RO means that the framework manages this variable and modules and methods shouldn't change it.


You can add additional tools to your mini OS using the directive:

# additional tools
PROGS=( "${PROGS[@]}" lsof parted fdisk cfdisk )

Relax-and-Recover can add its own GRUB menu entry to your local system's GRUB, which is convenient to restore a system without the need for an additional boot media. This only works if your system can (still) boot from the local disk and the disaster didn't destroy that disk.

Relax-and-Recover automatically detects and enables serial console support. This is extremely useful if the only way to access the console during disaster is a Java-based console riddled with keyboard bugs and slow screen refreshes.

Relax-and-Recover ships with a set of useful commands in its shell history in recover mode. This makes it possible to quickly look for a command to help troubleshoot, or modify an important file during recovery.

If the original system was configured to log on remotely through the use of SSH keys, Relax-and-Recover preserved those keys on the rescue environment and you can access the rescue environment from the network as you were used to before.

During recovery at any stage you can re-run Relax-and-Recover, modify the layout file for recreating the structure and intervene when restoring the backup.

During restore Relax-and-Recover uses the saved system layout as the basis for recreating a workable layout on your new system. If your new hardware is very different, it's advised to copy the layout file /var/lib/rear/layout/disklayout.conf to /etc/rear and modify it according to what is required.

cp /var/lib/rear/layout/disklayout.conf /etc/rear/
vi /etc/rear/disklayout.conf

Then restart the recovery process: rear recover. Relax-and-Recover translates this layout file into a shell procedure (/var/lib/rear/layout/ that contains all the needed instructions for recreating your desired layout.

HP StartARray controller presents some problems. You can find more information about your HP SmartArray setup by running one of the following commands:

# hpacucli ctrl all show detail
# hpacucli ctrl all show config
# hpacucli ctrl all show config detail

You can find these commands as part of the history of the Relax-and-Recover shell.


To use Relax-and-Recover you always call the main script /usr/sbin/rear:

# rear help

Usage: rear [-h|--help] [-V|--version] [-dsSv] [-D|--debugscripts SET] [-c DIR] [-r KERNEL] [--] COMMAND [ARGS...]

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at:

Available options:
 -h --help           usage information
 -c DIR              alternative config directory; instead of /etc/rear
 -d                  debug mode; log debug messages
 -D                  debugscript mode; log every function call (via 'set -x')
 --debugscripts SET  same as -d -v -D but debugscript mode with 'set -SET'
 -r KERNEL           kernel version to use; current: '3.12.49-3-default'
 -s                  simulation mode; show what scripts rear would include
 -S                  step-by-step mode; acknowledge each script individually
 -v                  verbose mode; show more output
 -V --version        version information

List of commands

  1.  checklayout     check if the disk layout has changed
  2.  dump            dump configuration and system information
  3.  format          format and label media for use with rear
  4.  mkbackup        create rescue media and backup system
  5.  mkbackuponly    backup system without creating rescue media
  6.  mkrescue        create rescue media only
  7.  recover         recover the system
  8.  validate        submit validation information
Use 'rear -v help' for more advanced commands.
To view/verify your configuration, run rear dump. It will print out the current settings for BACKUP and OUTPUT methods and some system information.

To create a new rescue environment, simply call rear mkrescue. Do not forget to copy the resulting rescue system away so that you can use it in the case of a system failure. Use rear mkbackup instead if you are using the built-in backup functions (like BACKUP=NETFS)

To recover your system, start the computer from the rescue system and run rear recover. Your system will be recovered and you can restart it and continue to use it normally.

Performing a System Rescue (from Red Hat documentation)

To perform a restore or migration:
  1. Boot the rescue system on the new hardware. For example, burn the ISO image to a DVD and boot from the DVD.
  2. In the console interface, select the "Recover" option:

    Figure 27.1. Rescue system: menu (omitted)

  3. You are taken to the prompt:

    Figure 27.2. Rescue system: prompt

    Warning: Once you have started recovery in the next step, it probably cannot be undone and you may lose anything stored on the physical disks of the system.

  4. Run the rear recover command to perform the restore or migration. The rescue system then recreates the partition layout and filesystems:

    Figure 27.3. Rescue system: running "rear recover"

  5. Restore user and system files from the backup into the /mnt/local/ directory.

    Example 27.3. Restoring User and System Files

    In this example, the backup file is a tar archive created per instructions in Section, "Configuring the Internal Backup Method". First, copy the archive from its storage, then unpack the files into /mnt/local/, then delete the archive:
    ~]# scp [email protected]:/srv/backup/rhel7/backup.tar.gz /mnt/local/
    ~]# tar xf /mnt/local/backup.tar.gz -C /mnt/local/
    ~]# rm -f /mnt/local/backup.tar.gz
    The new storage has to have enough space both for the archive and the extracted files.
  6. Verify that the files have been restored:
    ~]# ls /mnt/local/

    Figure 27.4. Rescue system: restoring user and system files from the backup

  7. Ensure that SELinux relabels the files on the next boot:
    ~]# touch /mnt/local/.autorelabel
    Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.
  8. Finish the recovery by entering exit. ReaR will then reinstall the boot loader. After that, reboot the system:

    Figure 27.5. Rescue system: finishing recovery

    Upon reboot, SELinux will relabel the whole filesystem. Then you will be able to log in to the recovered system.

Restoring RHEL-based system with SELinux enabled

IMPORTANT: If SELinux is enabled on your system you need to ensure that SELinux relabels the files on the next boot:
~]# touch /mnt/local/.autorelabel
Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.

Affordable compact (FIT or bar form factor) 128 GB USB 3.0 Flash drives

"FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.



USB 2.0

Top Visited
Past week
Past month


Old News ;-)

[Aug 21, 2020] ReaR- Backup and Recover your Linux server with confidence by Sreejith Anujan

The article discusses several scarios of using ReaR. Not much new or interesting but looks like OK overview of version 2.4
Aug 19, 2020 |
... ... ... Deploy ReaR on the server to be backed up

On the production server, install the rear , genisoimage , and syslinux packages. In RHEL, these packages are part of the base repository.

... ... ...

As suggested by the login banner, run rear recover to restore the system by connecting to the storage server. From there, retrieve the backup.tar.gz and restore it to the right destination with appropriate permissions.

RESCUE production: ~ # rear -v -d recover
... ... ...

...ReaR is an integral part of many Linux-based backup solutions. OpenStack and the Red Hat OpenStack Platform use ReaR for the undercloud and control plane backup and restore. Watch for future articles in this space addressing the patching and rollback options for applications and operating systems.

For more on using ReaR in a Red Hat Enterprise Linux production environment, be sure to consult this solution in the Red Hat Customer Portal.

Sreejith Anujan is a cloud technology professional with more than 14 years of experience in on-premise and public cloud providers. He enjoys working with customers on their enablement plans to upskill the technical team on container and automation tooling. More about me

[Mar 23, 2020] Relax-and-Recover - Backup and Recover a Linux System

A good overview
Mar 23, 2020 |


Relax-and-Recover Key Features:
  1. It has a modular design written in Bash and can be extended using custom functionality.
  2. Supports various boot media including ISO, PXE, OBDR tape, USB or eSATA storage.
  3. Supports a variety of network protocols including FTP, SFTP, HTTP, NFS, and CIFS for storage and backup .
  4. Supports disk layout implementation such as LVM, DRBD, iSCSI, HWRAID (HP SmartArray), SWRAID, multipathing, and LUKS (encrypted partitions and filesystems).
  5. Supports both third-party and internal backup tools including IBM TSM, HP DataProtector, Symantec NetBackup, Bacula; tar and rsync .
  6. Supports booting via PXE, DVD/CD, bootable tape or virtual provisioning.
  7. Supports a simulation model that shows what scripts are run without executing them.
  8. Supports consistent logging and advanced debugging options for troubleshooting purposes.
  9. It can be integrated with monitoring tools such as Nagios and Opsview.
  10. It can also be integrated with job schedulers such as cron .
  11. It also supports various virtualization technologies supported (KVM, Xen, VMware).

In this article, you will learn how to install and configure ReaR to create a rescue system and/or system backup using a USB stick and rescue or restore a bare-metal Linux system after a disaster.

... ... ...

[Apr 03, 2018] Relax and Recover

Apr 03, 2018 |

The main purpose of ReaR is to create a bootable image, based on what is currently installed on a Linux host, that can be used to partition disks and retrieve a backup of the system. There are options for where to create the bootable image and what to do with it after it has been created.

The bootable image can be a USB device, an ISO file or a number of other options.

If you create a bootable image on a USB device then you may also wish to create a backup of your system on the same device, which ReaR will support.

When creating a bootable image as an ISO file you have a multitude of options for what do to with the file in order to get it off the box so that it can be used for recovery. The two options I have used are rsync and TSM .

The misconception I mentioned earlier is the belief that ReaR will backup your system. It can do that, but it is not a given and depends on your configuration acheter du cialis 5 .

... ... ...

Example Procedure

The following is an example of the produce to protect a system with ReaR and TSM during some operating system patching activities (assumes TSM is already installed):

  1. Install ReaR (rpms are available here ).
  2. Configure ReaR to use TSM and to create an ISO file by updating /etc/rear/local.conf with a line of OUTPUT=ISO and another with BACKUP=TSM.
  3. Run "rear -v mkrescue" to create the bootable ISO and send it to TSM (mkbackup would have the same effect in this case as TSM will be handling the file system backups independently – I feel mkrescue makes it clearer what you're doing).
  4. Perform a incremental backup of your file systems with TSM using "dsmc inc ".
  5. Do your patching activity.

If all goes well then you don't need to boot from the ReaR ISO and restore you operating system. But, let's say it didn't go well. Your system will no longer boot and there's no immediately obvious way forward. You decide to restore. The procedure is:

  1. Restore the ReaR ISO to a location that will allow you to present it to the server. This is most likely to be your desktop so you can present the ISO file as a virtual CD-ROM over the ILOM interface.
  2. Present the ISO to the host to be recovered.
  3. Boot the host from the ISO – It is highly likely that you'll need to change the boot order or get a pop-up menu to select the ISO as the boot media.
  4. Select "Recover <hostname>" at the grub prompt.
  5. Log in as root (password not required).
  6. Run "rear -v recover" and answer the interactive prompts.


Since starting to use ReaR I have encountered two problems:

  1. When recovering a host that used an ext4 file system for /boot I found myself facing at message of "Error 16: Inconsistent filesystem structure." from grub. After a bit of digging around and trying to understand what the issue was I ended up modifying the /var/lib/rear/layout/disklayout.conf ReaR file to change the file system type for /boot from ext4 to ext2. I initially tried ext3, but as the system did not use ext3 for any of the file systems the module was not available.
  2. The version of ReaR that I was using had a bug ( tracked on GitHub ) that affected systems that do not have a separate /boot partition. There is a patch for the bug available, but if like me you're happy to have a manual workaround, you need to perform the following actions after the restore completes:
# chroot /mnt/local
# PATH=/bin:/sbin:/usr/bin
# grub-install <disk path>
# exit
# reboot

Finally, it's worth mentioning that ReaR is written in shell and is open source.

[Nov 05, 2016] Relax and Recover – How Did I Do That

21 August 2014

Start a backup on the CentOS machine

Add the following lines to /etc/rear/local.conf:

BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )

BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )

Now make a backup

[root@centos7 ~]# rear mkbackup -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
mkdir: created directory '/var/lib/rear/output'
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-centos7.iso (90M)
Copying resulting files to nfs location
Encrypting disabled
Creating tar archive '/tmp/rear.QnDt1Ehk25Vqurp/outputfs/centos7/2014-08-21-1548-F.tar.gz'
Archived 406 MiB [avg 3753 KiB/sec]OK
Archived 406 MiB in 112 seconds [avg 3720 KiB/sec]

Now look on your NFS server

You'll see all the files you'll need to perform the disaster recovery.

total 499M
drwxr-x- 2 root root 4.0K Aug 21 23:51 .
drwxr-xr-x 3 root root 4.0K Aug 21 23:48 ..
-rw--- 1 root root 407M Aug 21 23:51 2014-08-21-1548-F.tar.gz
-rw--- 1 root root 2.2M Aug 21 23:51 backup.log
-rw--- 1 root root 202 Aug 21 23:49 README
-rw--- 1 root root 90M Aug 21 23:49 rear-centos7.iso
-rw--- 1 root root 161K Aug 21 23:49 rear.log
-rw--- 1 root root 0 Aug 21 23:51 selinux.autorelabel
-rw--- 1 root root 277 Aug 21 23:49 VERSION

Author: masterdam79

You can also connect with me on Google+ View all posts by masterdam79

Author masterdam79/
Posted on 21 August 2014/

dheeraj says:

31 August 2016 at 02:26

is it possible to give list of directories or mount points while giving mkbackup to exclude from backup. Like giving a file with list of all directories that need to be excluded ??

masterdam79 says:

26 September 2016 at 21:50

Have a look at
Should be possible if you ask me.

[Nov 05, 2016] Relax-and-Recover – Freecode

Nov 05, 2016 |

Relax-and-Recover (Rear) is a bare metal disaster recovery and system migration solution, similar to AIX mksysb or HP-UX ignite. It is composed of a modular framework and ready-to-go workflows for many common situations to produce a bootable image and restore from backup using this image. It can restore to different hardware, and can therefore be used as a migration tool as well. It supports various boot media (including tape, USB, or eSATA storage, ISO, PXE, etc.), a variety of network protocols (including SFTP, FTP, HTTP, NFS, and CIFS), as well as a multitude of backup strategies (including IBM TSM, HP DataProtector, Symantec NetBackup, Bacula, and rsync). It was designed to be easy to set up, requires no maintenance, and is there to assist when disaster strikes. Recovering from disaster is made very straight-forward by a 2-step recovery process so that it can be executed by operational teams when required. When used interactively (e.g. when used for migrating systems), menus help make decisions to restore to a new (hardware) environment.

Release Notes: Integrated with duply/duplicity support. systemd support has been added. Various small fixes and improvements to tape support, Xen, PPC, Gentoo, Fedora, multi-arch, storage ... layout configuration, and serial console integration.


Release Notes: This release adds support for multipathing, adds several improvements to distribution backward compatibility, improves ext4 support, makes various bugfixes, migrates HWADDR ... after rescovery, and includes better systemd support.


Release Notes: Multi-system and multi-copy support on USB storage devices. Basic rsync backup support. More extensive exclude options. The new layout code is enabled by default. Support ... for Arch Linux. Improved multipath support. Experimental btrfs support.


Release Notes: Standardization of the command line. The default is quiet output; use the option -v for the old behavior. Boot images now have a comprehensive boot menu. Support for IPv6 ... addresses. Restoring NBU backup from a point in time is supported. Support for Fedora 15 (systemd) and RHEL6/SL6. Improved handling of HP SmartArray. Support for ext4 on RHEL5/SL5. Support for Xen paravirtualization. Integration with the local GRUB menu. Boot images can now be centralized through network transfers. Support for udev on RHEL4. Many small improvements and performance enhancements.


Release Notes: This release supports many recent distributions, including "upstart" (Ubuntu 7.10). It has more IA-64 support (RHEL5 only at the moment), better error reporting and catching, ... Debian packages (mkdeb), and improved TSM support.


[Nov 05, 2016] Affordable compact (FIT or bar form factor) 128 GB Flash drives

"FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.
Nov 05, 2016 |



USB 2.0

[Nov 04, 2016] Coding Style rear-rear Wiki

Reading rear sources is an interesting exercise. It really demonstrates attempt to use "reasonable' style of shell programming and you can learn a lot.
Nov 04, 2016 |

Relax-and-Recover is written in Bash (at least bash version 3 is needed), a language that can be used in many styles. We want to make it easier for everybody to understand the Relax-and-Recover code and subsequently to contribute fixes and enhancements.

Here is a collection of coding hints that should help to get a more consistent code base.

Don't be afraid to contribute to Relax-and-Recover even if your contribution does not fully match all this coding hints. Currently large parts of the Relax-and-Recover code are not yet in compliance with this coding hints. This is an ongoing step by step process. Nevertheless try to understand the idea behind this coding hints so that you know how to break them properly (i.e. "learn the rules so you know how to break them properly").

The overall idea behind this coding hints is:

Make yourself understood

Make yourself understood to enable others to fix and enhance your code properly as needed.

From this overall idea the following coding hints are derived.

For the fun of it an extreme example what coding style should be avoided:

#!/bin/bash for i in `seq 1 2 $((2*$1-1))`;do echo $((j+=i));done


Try to find out what that code is about - it does a useful thing.

Code must be easy to read Code should be easy to understand

Do not only tell what the code does (i.e. the implementation details) but also explain what the intent behind is (i.e. why ) to make the code maintainable.

Here the initial example so that one can understand what it is about:

#!/bin/bash # output the first N square numbers # by summing up the first N odd numbers 1 3 ... 2*N-1 # where each nth partial sum is the nth square number # see # this way it is a little bit faster for big N compared to # calculating each square number on its own via multiplication N=$1 if ! [[ $N =~ ^[0-9]+$ ]] ; then echo "Input must be non-negative integer." 1>&2 exit 1 fi square_number=0 for odd_number in $( seq 1 2 $(( 2 * N - 1 )) ) ; do (( square_number += odd_number )) && echo $square_number done

Now the intent behind is clear and now others can easily decide if that code is really the best way to do it and easily improve it if needed.

Try to care about possible errors

By default bash proceeds with the next command when something failed. Do not let your code blindly proceed in case of errors because that could make it hard to find the root cause of a failure when it errors out somewhere later at an unrelated place with a weird error message which could lead to false fixes that cure only a particular symptom but not the root cause.

Maintain Backward Compatibility

Implement adaptions and enhancements in a backward compatible way so that your changes do not cause regressions for others.

Dirty hacks welcome

When there are special issues on particular systems it is more important that the Relax-and-Recover code works than having nice looking clean code that sometimes fails. In such special cases any dirty hacks that intend to make it work everywhere are welcome. But for dirty hacks the above listed coding hints become mandatory rules:

For example a dirty hack like the following is perfectly acceptable:

# FIXME: Dirty hack to make it work # on "FUBAR Linux version 666" # where COMMAND sometimes inexplicably fails # but always works after at most 3 attempts # see # Retries should have no bad effect on other systems # where the first run of COMMAND works. COMMAND || COMMAND || COMMAND || Error "COMMAND failed."

Character Encoding

Use only traditional (7-bit) ASCII charactes. In particular do not use UTF-8 encoded multi-byte characters.

Text Layout Variables Functions Relax-and-Recover functions

Use the available Relax-and-Recover functions when possible instead of re-implementing basic functionality again and again. The Relax-and-Recover functions are implemented in various lib/* files .

test, [, [[, (( Paired parenthesis See also

Recommended Links

Google matched content

Softpanorama Recommended

Top articles


Top articles


Contract support and subscription from developers (Gratien D'haese (IT3 Consultants)

Presentations and videos on Youtube:

ReaR- Backup and Recover your Linux server with confidence - Enable Sysadmin (Red hat)

For more on using ReaR in a Red Hat Enterprise Linux production environment, be sure to consult this solution in the Red Hat Customer Portal.



Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy


War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes


Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law


Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site


The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: January, 04, 2021