Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)

Relax-and-Recover on RHEL

Complex, brittle (but still useful) bare metal backup with tar as the default backup method

News Disaster Recovery Recommended Links  FIT USB flash drives Missing backup horror stories Bare metal recovery of Linux systems Cloning systems, disks and partitions
Unix tape archiver (Tar) Tar options for bare metal recovery Baseliners Data deduplication Gzip Unix cpio Dump and restore
dd Remote backups using dd  Recovery of lost files using DD Mount a partition from dd disk image rsync Backing Up And Restoring Linux With SystemImager Perl Backup Scripts
Cron and Crontab commands HP Data Protector Dell DRAC and VFLASH Data deduplication Sysadmin Horror Stories Humor Etc

Introduction

Relax-and-Recover a bare metal disaster recovery and system migration software. It is a complex, somewhat brittle because of this complexity, and not well documented (as usual for open source packages). Red Hat provides really old version in RPM for RHEL6 -- 1.17 (in comparison RHEL 5.x uses version 2.0 ;-) Stable releases are available as tgz files from the Sourceforge Downloads page. Each  tar file contain the main script that should go into /usr/sbin, the set of directories in /usr/shar/rear as well as documentation and sample config file.  There is also a make file for installation.

At the core, ReaR is a a set of scripts written in BASH. Scripts themselves are well written, demonstrate a good BASH style and as such represent  ultimate documentation to the package, so they do compensate for the low quality of the documentation, as the package is designed for sysadmin and  they all know or should know bash.

Version 1.17 Red Hat supplies is broken for G7 version of HP servers: you can backup but you can't restore. Interestingly enough it does work for HP G6 servers. If you discover this situation one option is to try to upgrade to version 2.x. If version 1.17 is installed, the upgrade can be done by simply copying /usr/sbin/rear executable and /usr/sbin/share tree from tar of version 2.x.  Please not that version 2.x has three digit numeric prefixes for scripts, while scripts in version 1.x have a two digit prefix.  If this helps and you can restore the test server, you can write a script or make file to propagate those updates to all relevant servers.

The quality of console messages is rather low (they do not include the name of script that generated them and they often missed fatal errors. You can't rely on  them, but the quality of  log written to /var/log/rear is good. It is better to watch ReaR log then console messages. The package has a useful debugging mode (see Learning rear internals and debugging ) which can and should be used to learn more about this system. 

There are several YouTube videos about this package which helps to understand it:

By default ReaR uses GNU tar for actual backup.  The main advantage of using tar is that it is a well known by all sysadmins utility which is reliable.  Another advantage is  that you do not need to learn something new, as "stack overflow" for sysadmins is a real problem: too many, too complex packages to maintain.  And it increases the confidence that your backup will be usable when the disaster strike -- you can use it even if rear breaks during recovery due to some bugs. Just boot from RHEL ISO into rescue mode and you can perform restore. 

The key problem with the packages like Relax and Recover is that they create there own "universe" with complex, non-obvious rules. some of them require experimentation to understand; some can be understood only looking at the code. So outside simplistic usage (format/mkrescure/mkbackup troika with default /etc/rear/local.conf file) you are in Alice in Wonderland environment and often it is unclear whether ReaR are worth trying. But even such simple operation as format has gotchas -- you need to supply the device name, not the partition name to it (clear USB bias ;-). And it should be unmounted.

In other words Rear if far from AKM47 type simplicity and reliability of backup (while plain vanilla Tar is close ;-). And unless you are willing to look at the code (you should -- it is well written code; a rear example of a good style of Bash programming ;-) you might never fully understand the capabilities of this software (not that everybody need to; simplistic usage is OK for many). 

Still despite all those warts it can be used, as default configuration is very useful if it works and some more complex configuration can be debugged using try and error process. As linux does not have any built-in bare metal recovery utility (other then DD; but you face problem writing zeros on unused parts of filesystem before making backup with it in order to save space) you also do not have too many alternatives. The other important  thing  is that because it is well written BASH scripts you can learn a lot of useful staff  about BASH programming experimenting with it.  If you are willing to look at the code, of course.

While "added complexity" often kills such packages, the issues in bare metal recovery are complex and at least for training purposes this package is a good candidate. It allows you to understand what is involved and how  to solve certain problems inherent in bare metal backup. And what is important in simple cases it works as expected. So if you do not need to deviate from typical operations, for example, excluding some filesystems from the backup, you are OK and just need to test whether it is able to resore the system (which also is not given). 

In a simplified way Rear is equivalent to booting to the rescue  mode on your RHEL distribution DVD, restoring MBR (which should be a part of tarball anyway, most sysadmins write it into /root with dd), and then using a tar file on a USB drive to restore the system. But the devil is always in details.

Also the fact that at the core of  Relax-and-Recover is an already known by you utility and backup can be restored outside  Relax  and Recover framework if very positive. Again, if restore fails (as it does with the version shipped with RHEL 6.9 on HP Intel servers like DL580 G7) you still can use tarball for recovery. In other words all this edifice of configuration files and setting which make rear very convenient (and very brittle) can be thrown out in case of need.

The key idea of Relax-and-Recover is that it is possible to construct bootable image with mini OS on the USB drive using files already available on the harddrive of installed Linux system.  If you add to this bootable mini disk a tar archive of system directories and some scripts, you can restore the system from a USB drive.  But implementation is wrong. The right approach would be to copy the OS from the system that is backuped (which just 6GB or less)  as all the necessary drivers are present and this installation will definitely works. With the approach that Relax-and-Recover uses, the constructed bootable image sometimes work, and sometime (on HP servers) does not.  In the latter case you need to boot to the recovery option in the RHEL ISO and you will be  able to continue the restoration process. So bugs in the restore script and configuration is not the end. 

By default rear uses /tmp filesystem for creation of the backup image. If it is not sufficient you need to specify alternative location for the image for rear to work

Still when it work, the recovery image represents a pretty capable mini-OS. You can SSH to it, you can add utilities missing on the fly via SCP fro other servers,  you can  chroot the restored image. When it does not work the fastest way is to create a second USB with mini OS (no GUI) installed from DVD and use it instead (two USB ports are not the problem on most servers, even HP ;-). If some USB ports need to be used for keyboard and mouse use a USB hub.

You can't relax until you test whether the restore mode works on your server or not.  It can and does blue screen on some servers. You still have the tarball so this is just a nuisance, but it kills "relax" part ;-)

You can't relax until you test whether the restore mode works on your server or not. It can and does blue screen on some servers ;-)

When it works you have a bare metal recovery solution created using basic Linux components.  Of course you can always install minimal OS on the USB drive and create tar manually. On systems where relax and recover blue screen on recovery this in the next option to try. But it requires shutting the server down and rebooting it from installation DVD or remotely mounted ISO to create the image. You can also do it on the fly by copying the OS manually and then installing grub and modifying grub menu to appropriate harddrive. But this requires some knowledge of grub.

In bare metal recovery utilities the devil is in details and relax-and-restore managed to account for most important details right. That's why despite it fragility it is a valuable backup solution: 

Relax-and-Recover uses tar with the set of tar options and exclude files  to created the tarball which later would be used to restore the system. If backup is not too large it can be put on permanently inserted FIT form factor USB drive. The current upper size for this low profile form factor is 256 GB (SanDisk 256GB Ultra Fit USB 3.1 ).  The same upper limit has metal bar form factor (SanDisk Ultra Flair).  Larger size "bar" form factor like PNY Pro Elite 512GB scale to 512GB. That means that  you can backup on  flash drive systems up to 512GB of used space (assuming 50% compression). Max for flash drives is probably 1TB (Kingston DataTraveler HyperX Predator 1TB USB 3.0 Flash Drive ). But above 256GB it is better to use USB 3.0 External hard drives such as Samsung T1 or Samsung T3, which are twice cheaper.

Relax-and-Recover produces a bootable image (rescue image) + tar file (backup).  This image restore partitioning of the system, including LVM based partitioning. Once that is done it initiates a restore from backup. Restores to different size of harddrives and  slightly different hardware are possible. Relax-and-Recover can therefore be used:

Currently Relax-and-Recover supports various boot media (incl. ISO, PXE, OBDR tape, USB or eSATA storage), a large variety of network protocols (sftp, ftp, http, nfs, rsync,  cifs). USB and NFS are probably the most important of them (USB is generally limited to 128GB, so larger backups should go to NFS), the most transparent of them and should be used whenever possible.

It also can be used with pre-existing backup tools such as  Bacula,  HP DataProtector, IBM TSM,  Symantec NetBackup,  but that place it outside of its main advantages area -- simple and reliable bare-metal backup solution.  In this combination it is neither simple, nor reliable ;-)

Relax-and-Recover recovery process is brittle. The setup from RPM requires installation of additional RPMs.

 

The process of recovery

The most typical and the simplest setup for recovery with ReaR is when your backup is on bootable USB drive or on rather small ISO file.  You backup generally consist of two parts

To recover from a failed drive just boot from USB. you will be presented with  the menu which has two recover option: automatic and manual. Manual gives you far more flexibility.  The process is  described in some derail in instructions provided Red Hat for RHEL and it will work on its clones such as CENTOS. Please note that RHEL the recovery is complicated by existence of SE Linux, if it is enabled.

Choosing manual recovery option you get into to the command prompt of a pretty capable mini-OS. Networking is enabled and SSH daemon is running, so you can SSH to it, which is huge advantage if you are working with old ILO or DRAC with  split cursor mentality ;-)

As you have access to your network you can add any utilities you want but which are missing on the fly via SCP.

You can also chroot  the restored image and edit grub setting and reinstall grub.

When the recovery does not work probably because mini-OS created does not boot correctly, the  fastest workaround  to create a second USB with mini OS (no  GUI) installed from DVD and use it instead (two USB ports are not the problem on most servers, even HP). If you have insufficient  USB ports (some HP servers has just two)  you can always use a USB hub.

Tar archive with data located on USB disk. It can  moved to hard drive unpacked and used manually for any purpose you which, including recovery of some lost files. 

There are some interesting possibilities to use Relax-and-Recover for migrating systems to slightly different hardware (which is often the case if computer is damaged beyond repair, for example some chip on motherboard failed.  In this case Relax-and-Recover provides step-by-step menus which help make decisions to restore the OS and data in new server.  

As source code is available you can look into it and  experiment with running it via bash with  -x option. It also has logging and optionally "extended output" mode which helps to understand the concepts behind Relax-and-Recover, troubleshoot during initial configuration and help debug during integration.

Notes:

Required RHEL packages

Relax-and-Recover is written entirely in Bash and theoretically should does not require any external programs beside classic Unix utilities (cat, cut, sort, dd, grep, etc.). But this is not true. It has its own and pretty convoluted set of dependences with several packages required to make the  rescue system work.

This is where a good simple idea became perverted. Unless you are patient you might even abandon the installation of the package, as the process is now well documented. I spend several hours trying to install in of RHEL 6.8 until I figured out what to do. This is not how decent open source software should be documented and developers should be ashamed.

The installation on RHEL fails if the supplementary software channel is not enabled before you attempt the installation. But it fails in its own particular way. Software is installed successfully, you can format USB drive, but when you try to make backup you get them message that syslinux-extlinux package is missing.

Also it you download those packages (for batch installation on many servers) and try to install them using yum for some reason on RHEL 6.x you face incompatibility of libraries problem and installation fails.  Among packages you need

On RHEL 6.5-6.8 you can install rear using the commands

yum-config-manager --enable repository rhel-6-server-optional-rpms
yum install rear syslinux-extlinux genisoimage syslinux

Quick Start Guide

In almost all circumstances you have to configure two main settings and their parameters: The BACKUP  method and the OUTPUT  method.

The backup method defines, how your data was saved and whether Relax-and-Recover should backup your data as part of the mkrescue process or whether you use an external application, e.g. backup software to archive your data.

The output method defines how the rescue system is written to disk and how you plan to boot the failed computer from the rescue system.

See /usr/share/rear/conf/default.conf for an overview of the possible methods and their options. For example

BACKUP=NETFS
OUTPUT=ISO

ReaR uses the following four variables to define how it backup particular system:

For NETFS method you have to define at least three variables for backup to be successful

Useful additional variables depends of your BACKUP_URL. We will discuss then for each of three BACKUP_URLs we cover in this page. One general useful setting is:

NETFS_KEEP_OLD_BACKUP_COPY=y  # If you want to keep the previous backup archive

Another is excluded filesystems. We will discuss it later (see Excluding large filesystems from the backup )

Variant 1: Backup to USB (creation of bootable recovery disk)

This is the most popular initial configuration of ReaR. You should proceed  as following:

  1. Prepare your USB media. We assume that USB media is at /dev/sdb.

    Before formatting the  device should be unmounted. The current version of rear does not unmount it and simply fails with a cryptic message

    Again, you better check this. The device should unmounted before you can start (USB disk are typically mounted on insertion to /media folder)

    Like always the case with format existing data will be destroyed. Formatting is very slow and can take for 32GB flash drive ten minutes on USB 2.0 connection. The drive is formatted as Ext3 filesystem.

    /usr/sbin/rear format /dev/sdb # YOU NEED the DEVICE NAME NOT A PARTITION NAME, IT ALSO SHOULD BE CORRECT DEVICE ;)
    NOTES:
    1. Here you need device name, not the partition name /dev/sdb1
    2. You can format the drive yourself without using REAR. That's probably  the "best practice" with the current version.

    After you type Enter, Relax-and-Recover asks you to confirm that you want to format the device. If everything is correct and you do not accidentally specify you primary harddrive :-), then answer

    Yes

    The most common reason for this command to abort with error is that the device is still mounted.

    If operation is successful, the device will be labeled REAR-000 by the format workflow.  This is the default label and you change it via configuration file if you object to it.
     

  2. Edit the /etc/rear/local.conf configuration file. For example you can write initial set of pretty generic parameters using  cat:
    cat > /etc/rear/local.conf <<EOF
    ### write the rescue initramfs to USB and update the USB bootloader
    OUTPUT=USB
    
    ### create a backup using the internal NETFS method, using 'tar'
    BACKUP=NETFS
    
    ### write both rescue image and backup to the device labeled REAR-000
    BACKUP_URL=usb:///dev/disk/by-label/REAR-000
    EOF

    If you have multiple boxes you can distribute this file to other boxes: the content of this file is server independent. If you already have such a file on other server, you can scp it to the new server you are configuring.

    Typically you also can benefit from the exclusion of certain directories, but not at this stage of learning.
     

  3. Make a backup of your /boot directory (see below). Normally sysadmin usually have such a backup, but on some servers they might nor. If not just do it (see below)
  4. CHECK that your USB drive is mountable and is not READ ONLY.  That sometimes happens with USB drives that they switch to read-only mode  and in this case Rear might happily delete /boot directory on your system.

    You need to verify that your USB drive used for backup can be mounted and is writable.
    That's important as Rear misbehaves if USB drive can't be mounted

    View the log immediately after the operation you specified ended and browse it for error messages

  5. Create a rescue image. This is essentially a test run as this is pretty quick operation. The size of rescue image is about 100MB. We want verbose output (-v option).
    sudo /usr/sbin/rear -v mkrescue
    You will get output similar to the following:
    Relax-and-Recover <version>
    Using log file: /var/log/rear/rear-<hostname>.log
    Creating disk layout
    Creating root filesystem layout
    Copying files and directories
    Copying binaries and libraries
    Copying kernel modules
    Creating initramfs
    Writing MBR to /dev/sdb
    Copying resulting files to usb location

    If you can create rescue image that means your rear installation works at least in this part, but you need to restore it at least once to be sure that respore part is OK too.

    NOTE: if you forgot to create configuration file rear will write rescue ISO image to /var/lib/rear/output/rear-`hostname`.iso
     

  6. Check the log file for possible errors. This is a VERY important step and should not be skipped -- rear is a brittle software. 
    And it can play jokes with you. 
     

    Checking the log is a very important step and should not be skipped -- rear is a brittle software.

     And it can play jokes with you when you need to recover your server

    I would also recommend to use some "we do not care about server" and try to restore it using your version of ReaR to make sure that it does not contain catastrophic bugs. Because you can encounter one at the most inopportune moment. But the structure of rear backup is useful even if restore for a particular version is not working. And the backup tar archive is usable too. 

    That means that you can restore the system step by step yourself following the scripts rear provides.  All the necessary information is still present that that' the most important part of the whole exercise.
     

  7. Reboot your system and try to boot from the USB device. You should get a self-explaining boot menu. Verify that the server can boot itself after 30 sec delay. On HP 580 G7 with rear 1.17.2  I encountered a bug due to which it blow up with the message invalid opcode of something like that. You can try restore the OS if this is a test computer to ensure that the version of Relax and Recover you use does not contain some glaring errors. 
     
  8. If your USB device has enough space, initiate a backup.  Backup of filesystem containing, say 32GB on the flash drive is a long operation which take an hour or so so you need to submit it either in screen session (I experienced problems with running it with nohup prefix; also you will not see any output until it ends which is a bad idea) You can use the following command:
    /usr/sbin/rear -v mkbackup
Important:

As tar is gziped you get (depending on the content on you filesystems) approximately 2/3 smaller image the the size of filesystem. That means that 64GB flash can store backup of ~100GB data.  With 256GB FIT drive (the max size of thise type of derive as of December 2017) you probably can backup half a terabyte of data.

If you have more then that you need to use NFS of rsync for storing backup on remote server or use USB drive instead of flash drive. 5TB USB drive is now about $120 (Seagate USB drive at Amazon).

One of the positive things about Relax and Restore is that the rescue image configure internet for you so all shares that are available on "normal" server are accessible from the rescue image boot-up. It is not that difficult to configure network manually but it is done automatically for the you it is even better. 

After you have such an image  you are now better better prepared for catastrophic failure of harddrives or other server components.  But if server has large partitions you need to use NFS of rsync to put the image on a remote filesystem. that would be the next step in learning rear.

Variant 2: Backup to a local filesystem

You can put your backup on any locally mounted filesystem. Similar to previous, but you define in configuration OUTPUT=ISO and directory to store your backup in BACKUP_URL.

BACKUP=NETFS
OUTPUT=ISO
BACKUP_URL=file:///DIRECTRY_WITH_YOUR_BACKUP

Variant 3:  Backup using NFS

Of course, you can backup to any mounted NFS share using file method (see above), but in this case your backup will not be available during the restore operation as this filesystem will not be mounted. So in such cases you need to define BACKUP_URL in form that can be used during the restore operation and provide IP parameters to endure that this server is assessable.
BACKUP_URL=nfs://server/path
For example:
BACKUP_URL="nfs://192.168.122.1/nfs/rear/"

And since typically all servers use NTP for time synchronization, you should probably also add these lines to iether /etc/rear/local.conf or to  /etc/rear/site.conf as this setting is typically site wise:

TIMESYNC=NTP
During restore operation, you need to have access to correct network settings.  If you have DHCP on this segment you can use
USE_DHCLIENT=yes

Otherwise you need to ask Rear to save your static networking settings via the directive:

USE_STATIC_NETWORKING=y

Example: NFS Share (osbconf.org)

# Create ReaR rescue media as ISO image
OUTPUT=ISO
# optionally define backup software, e.g. TSM, NBU, DP, BACULA
BACKUP=NETFS
NETFS_URL=nfS://e6500osl23/srv/backup
NETFS_KEEP_OLD_BACKUP_COPY=Y
# the following is required on older VMware VMs
# MODULES_LOAD=( vmxnet )
# line below was automatically added by 21_include_dhclient.sh
DHCLIENT_BIN=dhcpcd
# line below was automatically added by 21_include_dhclient.sh
DHCLIENT6_BIN=
# Enable the use of cfg2html (set to non-empty to disable). cfg2html must be
installed independently from ReaR.
SKIP_CFG2HTML=N
# Warn about NETFS not being a professional backup solution
NETFS_SKIP_WARNING=Y
# additional tools
PROGS=( "${PROGS[@]}" lsof parted fdisk cfdisk )

Always check ReaR logs after the  backup

Always check your logs. The fact that you have not warnings on console does not means that your backup is successful

Always check your logs. The fact that you have not warnings on console does not means that your backup is successful

During backup to USB drive the most typical error is the situation when there is no space on the device left -- the size of the backup exceeds the size of the USB drive used.  If ReaR is aborted, it does not clean the space on the USB drive so you need to do it manually or subsequent backup will fail.

What exactly happened often in not clear and only analysis of the log file can tell you exect situation and point ot the reason

So the analysis of the log file is a must.

Creating your local and site configuration: local.conf vs site.conf

To configure Relax-and-Recover you have to edit two configuration files in /etc/rear/

Settings that are common to all your server in partcular segment of datacenter  should generally go to site.conf (which is intended to be distributed to all servers in your backup scheme)

Unique for a particular server setting should go to local.conf (which is intended  to capture unique settings which are necessary for some servers and which should not be propagated to other servers) . 

Settings in local.conf overwrite settings in site.conf.

All other configuration files hold defaults for various distributions and should not be changed.

In /usr/share/rear/conf/ there are also some configuration files which you can study. They have extension .conf 

The most useful for study is default.conf (which is also the largest). It is well commented and contains ton of variable that you can set (and hope that they work and you understand their semantics correctly, which is not given ;-). This "preoccupation with variables" is a weak point of ReaR, but we have what we have. 

Using the site.conf  does not makes any sense if you do not distribute it to all your systems. If you have identical configuration on all servers you can also distribute local.conf t o all servers instead.

Excluding large filesystems from the backup

Often you need to exclude some filesystems from backup (typically large data filesystems that are backuped separately via rsync, or some other method)

The most useful for study is /usr/share/rear/conf/default.conf

Source view -etc-rear-default.conf

# Custom command backup stuff
# BACKUP=EXTERNAL
# examples for external backup. In this mode your external program must do EVERYTHING
# In the example below we backup / to the vms host via tar and netcat
# NOTE: The EXTERNAL_* commands can be also defined as an array () to better protect
# arguments with blanks
# NOTE: The EXTERNAL_* commands will be run inside eval like this:
# eval "${EXTERNAL_BACKUP[@]}"
#
# Command to backup the required data
# This example saves the data via SSH to a remote system called vms
EXTERNAL_BACKUP="tar -c -l -z / | ssh vms 'cat >rear64/backup.tar.gz'"
# Command to restore the data
EXTERNAL_RESTORE="ssh vms cat rear64/backup.tar.gz | tar -C /mnt/local -x -z"
# Command to verify the availability of the backup resource, will be executed only if PING=1
# NOTE: This command will be run within the ProgressBar system !! You should therefore avoid
# output on STDOUT by rerouting that to FD 8, the progress bar
EXTERNAL_CHECK="ssh vms date 1>&8"
 
# How to exclude something ----- EXCLUDES -------
#
# You cannot exclude a device (e.g. /dev/sdg) directly. Instead you have to exclude everything
# ON that device and then the dependancy tracker will automaticall exclude the device from the
# recovery (because there won't be any recovery information for that "unnecessary" device).
#
# Furthermore, you have to exclude MD devices and LVM2 volume groups separately as there is no
# automatic detection of these dependancies (yet, please write and submit it !)
 
# Exclude filesystems by specifying their mountpoints. Will be automatically added to the
# $BACKUP_PROG_EXCLUDE array during backup to prevent the excluded filesystems' data to
# be backed up
# examples: /tmp
#           /media/bigdisk
EXCLUDE_MOUNTPOINTS=()
 
# Exclude MD devices
# examples: /dev/md0
#           /dev/md/0
EXCLUDE_MD=()
 
# Exclude LVM2 volume groups. This will automatically exclude also the creation of the corresponding
# physical and logical volumes that belong to the excluded volume group.
#
# NOTE: YOU MUST ALSO EXCLUDE THE CORRESPONDING MOUNTPOINTS IN EXCLUDE_MOUNTPOINTS (see above)
#       OTHERWISE THE RECOVERY ***WILL*** TRY TO RECREATE THE FILESYSTEMS ONTO NON-EXISTING LVs
#
#       Y O U   H A V E   B E E N   W A R N E D  ! ! !
EXCLUDE_VG=()

You can use option EXCLUDE_MOUNTPOINTS to exclude some mounting filesystems.  For example:

EXCLUDE_MOUNTPOINTS=( '/Usbdrive' '/Apps' '/Scratch' ) 

This is a BASH array which uspposldly is added to to "main exclusion array" which is BACKUP_PROG_EXCLUDE  (and which already contains sevral components such as /tmp, so you should not overwrite it)

BACKUP_PROG_EXCLUDE

You could also try running rear with the -d switch. In your /tmp/rear-XXX/ folder should be a backup-exclude.txt file. It's in tar exclude syntax. That's an interesting file to share. Using rear dump is not helpful as it does not include those parameters

Here is a note on this that I have found Excluding a folder in a rear backup · Issue #216 · rear-rear · GitHub

 
Closed
cocampbe opened this Issue on Mar 27 2013 · 13 comments
I've been perusing the docs and have not come up with a solution yet. I am trying to exclude a folder that contains a large amount of logs. Here is what I have in the local.conf.

ONLY_INCLUDE_VG=( VolGroup00 )
EXCLUDE_COMPONENTS=( "fs:/usr/openv/logs/nbftsrvr" )

So far this does not work. I am not sure if the order matters, or if I am using the wrong exclude directive. Any insight is greatly appreciated.

 

jhoekx on Mar 27 2013

Member
EXCLUDE_COMPONENTS only works for real system components like filesystems or logical volumes. If you just want to exclude a directory you have to use the BACKUP_PROG_EXCLUDE array.

cocampbe on Mar 27 2013

Cool. I will do that. I was about to try that, but I just wasn't sure. I also thought about just trying EXCLUDE_MOUNTPOINTS even though it technically is not.
 

jhoekx on Mar 27 2013

Member
Can you show us the local.conf you're using? To make sure the BACKUP_PROG_EXCLUDE syntax is OK.

cocampbe on Mar 27 2013

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL="nfs://some_server/rear/"
NETFS_KEEP_OLD_BACKUP_COPY=1
ONLY_INCLUDE_VG=( VolGroup00 )
BACKUP_PROG_EXCLUDE=( /usr/openv/logs/nbftsrvr )

cocampbe on Mar 27 2013

I just added single quotes to BACKUP_PROG_EXCLUDE=( '/usr/openv/logs/nbftsrvr' ). trying again.

jhoekx on Mar 27 2013

Member
The line is not entirely correct. It should be:
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/usr/openv/logs/nbftsrvr' )

You could also try running rear with the -d switch. In you /tmp/rear-XXX/ folder should be a backup-exclude.txt file. It's in tar exclude syntax. That's an interesting file to share.

 

cocampbe on Mar 27 2013

I checked the file and it contains /usr/openv/logs/nbftsrvr.
 

jhoekx on Mar 27 2013

Member
The file is sent to tar as-is. Maybe try /usr/openv/logs/nbftsrvr/*?

cocampbe on Mar 27 2013

File is now showing /usr/openv/logs/nbftsrvr/*. I am running rear again.

cocampbe on Mar 27 2013

I just tailed the backup.log file and saw that it is not copying those files. So it is working. Looks like I may be backing up something else that is making the archive rather large. Thanks for your help.

cocampbe on Mar 27 2013

Yep. even though I have ONLY_INCLUDE_VG=( VolGroup00 ), it was backing up a san attached volume. I added it to the exclude list.

rear-06-layout-configuration.adoc at master · rear-rear · GitHub

Manual excludes

It seems prudent to prevent the external drives from ever being backed-up or overwritten. The default configuration contains these lines:
# Exclude components from being backed up, recreation information is active
EXCLUDE_BACKUP=()

# Exclude components during component recreation
# will be added to EXCLUDE_BACKUP (it is not backed up)
# recreation information gathered, but commented out
EXCLUDE_RECREATE=()

# Exclude components during the backup restore phase
# only used to exclude files from the restore.
EXCLUDE_RESTORE=()
To prevent an inadvertently mounted backup filesystem being added to the restore list, the easiest way is to add the filesystem to the EXCLUDE_RECREATE array.
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/media/backup" )

 

Road to hell is paved with good intentions: architectural flaws on Rear

While the motivation of the creators of this backup software were noble, the net result is somewhat questionable because complexity of the resulting scripts bite them in the back.  The architecture chosen (complex scripts driven by million of env variables) is questionable and ability to correct errors in generated commands (for example in the tar commend used for the backup) is limited.  There are also complex interdependencies  and not all situations  are diagnosed on the console

In pretty common case when that tarball overflow USB drive, there no explicit diagnostics about this important event on the console and if you do not check logs you can be royally screwed. 

When you exceed certain level of complexity with bash scripts  the resulting "mess" is not only unmanageable, it is outright dangerous. Unless you are happy with the default "mkrestore" option, you can be royally screwed (especially if you fail to look at the logs). Backup can fail due to the lack of space on, say, USB drive, but there will be no diagnostic messages. Or backup can succeed, but the content of the tarball will be not what you expect. As the result, if you do not  check that content of the tarball beforehand, you can destroy some filesystems during restore operation by restoring filesystems that you did not intent to restore. There are several other horror stories that can happen, when the complexity of the scripts far exceed you level of understanding and generated command  are not explicitly listed.

Here is one example. In most real cases you need to exclude some filesystems and tar provides this capability. But with the maze of scripts in ReaR this capability is not available directly in ReaR.  You need to populate a special array EXCLUDE_MOUNTPOINTS :

EXCLUDE_MOUNTPOINTS=( '/Usbdrive' '/Apps' '/Scratch' ) 

You can't provide you own exclude file, you can't correct generated command with you own script, unless you want to mess with the source code. And you need to rely on variable provided by ReaR, which depends on the processing logic of corresponding script so without looking at the script code you will be trying to find a black can in dark room.  The logic in the script that govern processing of this array 40_create_include_exclude_files.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# backup all local filesystems, as defined in mountpoint_device

for k in "${BACKUP_PROG_INCLUDE[@]}" ; do
	test "$k" && echo "$k"
done > $TMP_DIR/backup-include.txt
# add the mountpoints that will be recovered to the backup include list
while read mountpoint device junk ; do
	if ! IsInArray "$mountpoint" "${EXCLUDE_MOUNTPOINTS[@]}" ; then
		echo "$mountpoint"
	fi
done <"$VAR_DIR/recovery/mountpoint_device" >> $TMP_DIR/backup-include.txt

# exclude list
for k in "${BACKUP_PROG_EXCLUDE[@]}" ; do
	test "$k" && echo "$k"
done > $TMP_DIR/backup-exclude.txt
# add also the excluded mount points to the backup exclude list
for k in "${EXCLUDE_MOUNTPOINTS[@]}" ; do
	test "$k" && echo "$k/"
done >> $TMP_DIR/backup-exclude.txt

Note that the script adds a backlash to each  excluded mount point at line 19.

Generally this idea of using approximately a hundred variables (yes around a hundred) that guide the behaviour of ReaR sucks. To add the insult to injury this capability is not well documented and the logic is too simplistic and requires some additional work

If you put additional slash at the end or make some other minor  mistake the configuration will not work, but no errors will be generated. The result can be checked  in  backup-exclude.txt file in tmp directory used by ReaR and it it is not right you can cancel the backup job immediately.  Here is the list of file in this directory (/tmp/rear.8T1ZsZOFSXWwA6g/tmp/ in our case)

-rwxrwx---+ 1 nnb None 35646232 Apr  3 10:08 backup.log
-rwxrwx---+ 1 nnb None       41 Apr  3 05:37 backup-exclude.txt
-rwxrwx---+ 1 nnb None       18 Apr  3 05:37 backup-include.txt
drwxrwx---+ 1 nnb None        0 Apr  3 16:51 boot
-rwxrwx---+ 1 nnb None       43 Apr  3 05:36 bootloader
-rwxrwx---+ 1 nnb None       28 Apr  3 05:36 copy-as-is-exclude
-rwxrwx---+ 1 nnb None   179899 Apr  3 05:36 copy-as-is-filelist
-rwxrwx---+ 1 nnb None 82642591 Apr  3 05:37 initrd.cgz
drwxrwx---+ 1 nnb None        0 Apr  3 16:51 mappings
-rwxrwx---+ 1 nnb None       93 Apr  3 05:36 parted
-rwxrwx---+ 1 nnb None       31 Apr  3 05:36 partitions
-rwxrwx---+ 1 nnb None       18 Apr  3 05:36 partitions_unsorted
-rwxrwx---+ 1 nnb None       18 Apr  3 05:36 partitions-data

The default content is as following

/tmp/*
/dev/shm/*
/var/lib/rear/output/*

Unlike the default content, all generated entries does not have * at the end, only a slash is added, for example

/Usbdrive/

If you did something wrong it will be visible in log stored in temp directory (/tmp/rear.8T1ZsZOFSXWwA6g/tmp/ in our case)  in the structure of tar command and in this case you need to cansel the backup and change the condfiuftaion so that it works as intended

here is an example of botched tar command which included "undesirable" /Usbdrive filesystem into the backup:

        2018-04-03 05:37:48.851184722

tar --warning=no-xdev --sparse --block-number --totals --verbose --no-wildcards-match-slash \ 
  --one-file-system --ignore-failed-read --anchored --gzip -X /tmp/rear.8T1ZsZOFSXWwA6g/tmp/backup-exclude.txt \
  -C / -c -f - /home / /Usbdrive /var/log/rear/rear-z17.log \
  | cat BACKUP_PROG_CRYPT_KEY | dd of=/tmp/rear.8T1ZsZOFSXWwA6g/outputfs/rear/z17/20180403.0536/backup.tar.gz

Tar option -C means  change to directory DIR,  so this command is running from the root directory.  You can generally test that command separately (in this case you do not need to pipe it to dd of course)

At this point your enthusiasm about this package is probably somewhat deflated. But I would like to stress that  mkrecover option is very useful and works.  And you get the tar command that failed,  modify it  and rerun getting the tarball you need.  That's the advantage of using  tar, although architectural decisions taken in ReaR are simplistic (usage of huge number of env variable for driving the scripts) are definitely wrong and the package is suffering from overcomplexity.

For more complex  cases you do need to know bash well to be able to debut  this beast.

Another problem is that ReaR does not  allow just generate commands it uses, which would allow correction of some subtle errors by your own scripts.  But the fact that you can extract the generated tar command from the log is hugely positive and greatly simplifies the debugging.

You can also run you own tar command based on the example you have found in the log instead of mkbackup and incorporate such a command in your own script

This is a BASH array which supposedly is added to to "main exclusion array" which is BACKUP_PROG_EXCLUDE  (and which already contains sevral components such as /tmp, so you should not overwrite it)

Rear cron job which is installed by RHEL package

Rear has a cron job installed when RHEL package is installed. It adds the file rear to the /etc/cron.d.  directory

The idea is to update rescue image automatically each time the layout of the box has been changed (for example you updated the version of OS) which is a good idea: 

30 1 * * * root /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue

Rear layout in RHEL

ReaR RPM is more or less LSB complaint and puts files in expected places:

It uses several locations during  backup process linuxtag.org

mkbackup method: /usr/share/rear/...

• conf/ - configuration files (/etc/rear/*.conf read last)

• prep/ - preparation work; checking the environment

• layout/save/ - save the disk layout /var/lib/rear/layout

• rescue/ - modules, network, storage,...

• build/ - populate the initial ramdisk for our rescue image

• pack/ - create the initrd and copy] kernel

• output/ - create the ISO image and copy to OUTPUTJJRL

• backup/ - make the backup archive to BACKUPJJRL

Learning rear internals and debugging

There is two presentations that help you to understand how to experiment with ReaR:

The simplest way is to create two virtual machines and backup on one and recover on the another.

There are several options that help you with the debugging:

To view/verify your configuration, run rear dump. It will print out the current settings for BACKUP  and OUTPUT  methods and some system information.

Relax-and-Recover is built as a modular framework. A call of rear <command> will invoke the following general workflow:

  1. Configuration: Collect system information to assemble a correct configuration (default, arch, OS, OS_ARCH, OS_VER, site, local). See the output of rear dump for an example. + Read config files for the combination of system attributes. Always read 'default.conf' first and 'site.conf', 'local.conf' last.
  2. Create work area in '/tmp/rear.$$/' and start logging to '/var/log/rear/rear-hostname.log'
  3. Run the workflow script for the specified command: '/usr/share/rear/lib/<command>-workflow.sh'
  4. Cleanup work area

The result of the analysis is written into configuration files under '/etc/rear/recovery/'. This directory is copied together with the other Relax-and-Recover directories onto the rescue system where the same framework runs a different workflow - the recovery workflow.

The recovery workflow is triggered by the fact that the root filesystem is mounted in a ram disk or tmpfs. Alternatively a "demo" recovery workflow can be started manually. This will simply recover all data into a subdirectory and not touch the hard disks (Phase 2).

The recovery workflow consists of these parts (identically named modules are indeed the same):

  1. Config: By utilizing the same configuration module, the same configuration variable are available for the recovery, too. This makes writing pairs of backup/restore modules much easier.
  2. Verify: Verify the integrity and sanity of the recovery data and check the hardware found to determine, whether a recovery will be likely to succeed. If not, then we abort the workflow so as not to touch the hard disks if we don’t believe that we would manage to successfully recover the system on this hardware.
  3. Recreate: Recreate the FS layout (partitioning, LVM, raid, filesystems, …​) and mount it under /mnt/local
  4. Restore: Restore files and directories from the backup to '/mnt/local/'. This module is the analog to the Backup module
  5. Finalize: Install boot loader, finalize system, dump recovery log onto '/var/log/rear/' in the recovered system.

FS layout

Relax-and-Recover tries to be as much LSB compliant as possible. Therefore ReaR will be installed into the usual locations:

/etc/rear/
Configurations
/usr/sbin/rear
Main program
/usr/share/rear/
Internal scripts
/tmp/rear.$$/
Build area

Layout of /etc/rear

default.conf
Default configuration - will define EVERY variable with a sane default setting. Serves also as a reference for the available variables 'site.conf' site wide configuration (optional)
local.conf
local machine configuration (optional)
$(uname -s)-$(uname -i).conf
architecture specific configuration (optional)
$(uname -o).conf
OS system (e.g. GNU/Linux.conf) (optional)
$OS/$OS_VER.conf
OS and OS Version specific configuration (optional)
templates/
Directory to keep user-changeable templates for various files used or generated
templates/PXE_per_node_config
template for pxelinux.cfg per-node configurations
templates/CDROM_isolinux.cfg
isolinux.cfg template
templates/…​
other templates as the need arises
recovery/…​
Recovery information

Layout of /usr/share/rear

skel/default/
default rescue FS skeleton
skel/$(uname -i)/
arch specific rescue FS skeleton (optional)
skel/$OS_$OS_VER/
OS-specific rescue FS skeleton (optional)
skel/$BACKUP/
Backup-SW specific rescue FS skeleton (optional)
skel/$OUTPUT/
Output-Method specific rescue FS skeleton (optional)
lib/*.sh
function definitions, split into files by their topic
prep/default/*.sh
prep/$(uname -i)/*.sh
prep/$OS_$OS_VER/*.sh
prep/$BACKUP/*.sh
prep/$OUTPUT/*.sh
Prep scripts. The scripts get merged from the applicable directories and executed in their alphabetical order. Naming conventions are: + _name.sh + where 00 < < 99
layout/compare/default/
layout/compare/$OS_$OS_VER/
Scripts to compare the saved layout (under /var/lib/rear/layout/) with the actual situation. This is used by workflow rear checklayout and may trigger a new run of rear mkrescue or rear mkbackup
layout/precompare/default/
layout/precompare/$OS_$OS_VER/
layout/prepare/default/
layout/prepare/$OS_$OS_VER/
layout/recreate/default/
layout/recreate/$OS_$OS_VER/
layout/save/default/
layout/save/$OS_$OS_VER/
Scripts to capture the disk layout and write it into /var/lib/rear/layout/ directory
rescue/…​
Analyse-Rescue scripts: …​
build/…​
Build scripts: …​
pack/…​
Pack scripts: …​
backup/$BACKUP/*.sh
Backup scripts: …​
output/$OUTPUT/*.sh
Output scripts: …​
verify/…​
Verify the recovery data against the hardware found, whether we can successfully recover the system
recreate/…​
Recreate file systems and their dependancies
restore/$BACKUP/…​
Restore data from backup media
finalize/…​
Finalization scripts

Inter-module communication

The various stages and modules communicate via standardized environment variables:

NAME TYPE Descriptions Example
CONFIG_DIR STRING (RO) Configuration dir '/etc/rear/'
SHARE_DIR STRING (RO) Shared data dir '/usr/share/rear/'
BUILD_DIR STRING (RO) Build directory '/tmp/rear.$$/'
ROOTFS_DIR STRING (RO) Root FS directory for rescue system '/tmp/rear.$$/initrd/'
TARGET_FS_ROOT STRING (RO) Directory for restore '/mnt/local'
PROGS LIST Program files to copy bash ip route grep ls …​
MODULES LIST Modules to copy af_unix e1000 ide-cd …​
COPY_AS_IS LIST Files (with path) to copy as-is '/etc/localtime' …​

RO means that the framework manages this variable and modules and methods shouldn't change it.

Tips

You can add adtional tools to your mini OS using the directive:

# additional tools
PROGS=( "${PROGS[@]}" lsof parted fdisk cfdisk )

Relax-and-Recover can add its own GRUB menu entry to your local system's GRUB, which is convenient to restore a system without the need for an additional boot media. This only works if your system can (still) boot from the local disk and the disaster didn't destroy that disk.

Relax-and-Recover automatically detects and enables serial console support. This is extremely useful if the only way to access the console during disaster is a Java-based console riddled with keyboard bugs and slow screen refreshes.

Relax-and-Recover ships with a set of useful commands in its shell history in recover mode. This makes it possible to quickly look for a command to help troubleshoot, or modify an important file during recovery.

If the original system was configured to log on remotely through the use of SSH keys, Relax-and-Recover preserved those keys on the rescue environment and you can access the rescue environment from the network as you were used to before.

During recovery at any stage you can re-run Relax-and-Recover, modify the layout file for recreating the structure and intervene when restoring the backup.

During restore Relax-and-Recover uses the saved system layout as the basis for recreating a workable layout on your new system. If your new hardware is very different, it's advised to copy the layout file /var/lib/rear/layout/disklayout.conf to /etc/rear and modify it according to what is required.

cp /var/lib/rear/layout/disklayout.conf /etc/rear/
vi /etc/rear/disklayout.conf

Then restart the recovery process: rear recover. Relax-and-Recover translates this layout file into a shell procedure (/var/lib/rear/layout/diskrestore.sh) that contains all the needed instructions for recreating your desired layout.

HP StartARray controller presents some problems. You can find more information about your HP SmartArray setup by running one of the following commands:

# hpacucli ctrl all show detail
# hpacucli ctrl all show config
# hpacucli ctrl all show config detail

You can find these commands as part of the history of the Relax-and-Recover shell.

Options

To use Relax-and-Recover you always call the main script /usr/sbin/rear:

# rear help

Usage: rear [-h|--help] [-V|--version] [-dsSv] [-D|--debugscripts SET] [-c DIR] [-r KERNEL] [--] COMMAND [ARGS...]

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Available options:
 -h --help           usage information
 -c DIR              alternative config directory; instead of /etc/rear
 -d                  debug mode; log debug messages
 -D                  debugscript mode; log every function call (via 'set -x')
 --debugscripts SET  same as -d -v -D but debugscript mode with 'set -SET'
 -r KERNEL           kernel version to use; current: '3.12.49-3-default'
 -s                  simulation mode; show what scripts rear would include
 -S                  step-by-step mode; acknowledge each script individually
 -v                  verbose mode; show more output
 -V --version        version information

List of commands

  1.  checklayout     check if the disk layout has changed
  2.  dump            dump configuration and system information
  3.  format          format and label media for use with rear
  4.  mkbackup        create rescue media and backup system
  5.  mkbackuponly    backup system without creating rescue media
  6.  mkrescue        create rescue media only
  7.  recover         recover the system
  8.  validate        submit validation information
Use 'rear -v help' for more advanced commands.
To view/verify your configuration, run rear dump. It will print out the current settings for BACKUP and OUTPUT methods and some system information.

To create a new rescue environment, simply call rear mkrescue. Do not forget to copy the resulting rescue system away so that you can use it in the case of a system failure. Use rear mkbackup instead if you are using the built-in backup functions (like BACKUP=NETFS)

To recover your system, start the computer from the rescue system and run rear recover. Your system will be recovered and you can restart it and continue to use it normally.

Performing a System Rescue (from Red Hat documentation)

To perform a restore or migration:
  1. Boot the rescue system on the new hardware. For example, burn the ISO image to a DVD and boot from the DVD.
  2. In the console interface, select the "Recover" option:

    Figure 27.1. Rescue system: menu

  3. You are taken to the prompt:

    Figure 27.2. Rescue system: prompt

    Warning: Once you have started recovery in the next step, it probably cannot be undone and you may lose anything stored on the physical disks of the system.

  4. Run the rear recover command to perform the restore or migration. The rescue system then recreates the partition layout and filesystems:

    Figure 27.3. Rescue system: running "rear recover"

  5. Restore user and system files from the backup into the /mnt/local/ directory.

    Example 27.3. Restoring User and System Files

    In this example, the backup file is a tar archive created per instructions in Section 27.2.1.1, "Configuring the Internal Backup Method". First, copy the archive from its storage, then unpack the files into /mnt/local/, then delete the archive:
    ~]# scp root@192.168.122.7:/srv/backup/rhel7/backup.tar.gz /mnt/local/
    ~]# tar xf /mnt/local/backup.tar.gz -C /mnt/local/
    ~]# rm -f /mnt/local/backup.tar.gz
    The new storage has to have enough space both for the archive and the extracted files.
  6. Verify that the files have been restored:
    ~]# ls /mnt/local/

    Figure 27.4. Rescue system: restoring user and system files from the backup

  7. Ensure that SELinux relabels the files on the next boot:
    ~]# touch /mnt/local/.autorelabel
    Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.
  8. Finish the recovery by entering exit. ReaR will then reinstall the boot loader. After that, reboot the system:

    Figure 27.5. Rescue system: finishing recovery

    Upon reboot, SELinux will relabel the whole filesystem. Then you will be able to log in to the recovered system.

Restoring RHEL-based system with SELinux enabled

IMPORTANT: If SELinux is enabled on your system you need to ensure that SELinux relabels the files on the next boot:
~]# touch /mnt/local/.autorelabel
Otherwise you may be unable to log in the system, because the /etc/passwd file may have the incorrect SELinux context.

Affordable compact (FIT or bar form factor) 128 GB Flash drives

"FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.

USB 3.0 FIT

USB 3.0 BAR

USB 2.0


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Apr 03, 2018] Relax and Recover

Apr 03, 2018 | blog.nasmart.me

The main purpose of ReaR is to create a bootable image, based on what is currently installed on a Linux host, that can be used to partition disks and retrieve a backup of the system. There are options for where to create the bootable image and what to do with it after it has been created.

The bootable image can be a USB device, an ISO file or a number of other options.

If you create a bootable image on a USB device then you may also wish to create a backup of your system on the same device, which ReaR will support.

When creating a bootable image as an ISO file you have a multitude of options for what do to with the file in order to get it off the box so that it can be used for recovery. The two options I have used are rsync and TSM .

The misconception I mentioned earlier is the belief that ReaR will backup your system. It can do that, but it is not a given and depends on your configuration acheter du cialis 5 .

... ... ...

Example Procedure

The following is an example of the produce to protect a system with ReaR and TSM during some operating system patching activities (assumes TSM is already installed):

  1. Install ReaR (rpms are available here ).
  2. Configure ReaR to use TSM and to create an ISO file by updating /etc/rear/local.conf with a line of OUTPUT=ISO and another with BACKUP=TSM.
  3. Run "rear -v mkrescue" to create the bootable ISO and send it to TSM (mkbackup would have the same effect in this case as TSM will be handling the file system backups independently – I feel mkrescue makes it clearer what you're doing).
  4. Perform a incremental backup of your file systems with TSM using "dsmc inc ".
  5. Do your patching activity.

If all goes well then you don't need to boot from the ReaR ISO and restore you operating system. But, let's say it didn't go well. Your system will no longer boot and there's no immediately obvious way forward. You decide to restore. The procedure is:

  1. Restore the ReaR ISO to a location that will allow you to present it to the server. This is most likely to be your desktop so you can present the ISO file as a virtual CD-ROM over the ILOM interface.
  2. Present the ISO to the host to be recovered.
  3. Boot the host from the ISO – It is highly likely that you'll need to change the boot order or get a pop-up menu to select the ISO as the boot media.
  4. Select "Recover <hostname>" at the grub prompt.
  5. Log in as root (password not required).
  6. Run "rear -v recover" and answer the interactive prompts.

Issues

Since starting to use ReaR I have encountered two problems:

  1. When recovering a host that used an ext4 file system for /boot I found myself facing at message of "Error 16: Inconsistent filesystem structure." from grub. After a bit of digging around and trying to understand what the issue was I ended up modifying the /var/lib/rear/layout/disklayout.conf ReaR file to change the file system type for /boot from ext4 to ext2. I initially tried ext3, but as the system did not use ext3 for any of the file systems the module was not available.
  2. The version of ReaR that I was using had a bug ( tracked on GitHub ) that affected systems that do not have a separate /boot partition. There is a patch for the bug available, but if like me you're happy to have a manual workaround, you need to perform the following actions after the restore completes:
# chroot /mnt/local
# PATH=/bin:/sbin:/usr/bin
# grub-install <disk path>
# exit
# reboot

Finally, it's worth mentioning that ReaR is written in shell and is open source.

[Nov 05, 2016] Relax and Recover – How Did I Do That

www.howdididothat.info

21 August 2014

Start a backup on the CentOS machine

Add the following lines to /etc/rear/local.conf:

OUTPUT=ISO
BACKUP=NETFS
BACKUP_TYPE=incremental
BACKUP_PROG=tar
FULLBACKUPDAY="Mon"
BACKUP_URL="nfs://NFSSERVER/path/to/nfs/export/servername"
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"

OUTPUT=ISO
BACKUP=NETFS
BACKUP_TYPE=incremental
BACKUP_PROG=tar
FULLBACKUPDAY="Mon"
BACKUP_URL="nfs://NFSSERVER/path/to/nfs/export/servername"
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"


Now make a backup

[root@centos7 ~]# rear mkbackup -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
mkdir: created directory '/var/lib/rear/output'
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-centos7.iso (90M)
Copying resulting files to nfs location
Encrypting disabled
Creating tar archive '/tmp/rear.QnDt1Ehk25Vqurp/outputfs/centos7/2014-08-21-1548-F.tar.gz'
Archived 406 MiB [avg 3753 KiB/sec]OK
Archived 406 MiB in 112 seconds [avg 3720 KiB/sec]

Now look on your NFS server

You'll see all the files you'll need to perform the disaster recovery.

total 499M
drwxr-x- 2 root root 4.0K Aug 21 23:51 .
drwxr-xr-x 3 root root 4.0K Aug 21 23:48 ..
-rw--- 1 root root 407M Aug 21 23:51 2014-08-21-1548-F.tar.gz
-rw--- 1 root root 2.2M Aug 21 23:51 backup.log
-rw--- 1 root root 202 Aug 21 23:49 README
-rw--- 1 root root 90M Aug 21 23:49 rear-centos7.iso
-rw--- 1 root root 161K Aug 21 23:49 rear.log
-rw--- 1 root root 0 Aug 21 23:51 selinux.autorelabel
-rw--- 1 root root 277 Aug 21 23:49 VERSION


Author: masterdam79

You can also connect with me on Google+ View all posts by masterdam79


Author masterdam79/
Posted on 21 August 2014/

dheeraj says:

31 August 2016 at 02:26


is it possible to give list of directories or mount points while giving mkbackup to exclude from backup. Like giving a file with list of all directories that need to be excluded ??

masterdam79 says:

26 September 2016 at 21:50

Have a look at https://github.com/rear/rear/issues/216
Should be possible if you ask me.

[Nov 05, 2016] Relax-and-Recover – Freecode

Nov 05, 2016 | freecode.com

Relax-and-Recover (Rear) is a bare metal disaster recovery and system migration solution, similar to AIX mksysb or HP-UX ignite. It is composed of a modular framework and ready-to-go workflows for many common situations to produce a bootable image and restore from backup using this image. It can restore to different hardware, and can therefore be used as a migration tool as well. It supports various boot media (including tape, USB, or eSATA storage, ISO, PXE, etc.), a variety of network protocols (including SFTP, FTP, HTTP, NFS, and CIFS), as well as a multitude of backup strategies (including IBM TSM, HP DataProtector, Symantec NetBackup, Bacula, and rsync). It was designed to be easy to set up, requires no maintenance, and is there to assist when disaster strikes. Recovering from disaster is made very straight-forward by a 2-step recovery process so that it can be executed by operational teams when required. When used interactively (e.g. when used for migrating systems), menus help make decisions to restore to a new (hardware) environment.

Release Notes: Integrated with duply/duplicity support. systemd support has been added. Various small fixes and improvements to tape support, Xen, PPC, Gentoo, Fedora, multi-arch, storage ... layout configuration, and serial console integration.

(more)

Release Notes: This release adds support for multipathing, adds several improvements to distribution backward compatibility, improves ext4 support, makes various bugfixes, migrates HWADDR ... after rescovery, and includes better systemd support.

(more)

Release Notes: Multi-system and multi-copy support on USB storage devices. Basic rsync backup support. More extensive exclude options. The new layout code is enabled by default. Support ... for Arch Linux. Improved multipath support. Experimental btrfs support.

(more)

Release Notes: Standardization of the command line. The default is quiet output; use the option -v for the old behavior. Boot images now have a comprehensive boot menu. Support for IPv6 ... addresses. Restoring NBU backup from a point in time is supported. Support for Fedora 15 (systemd) and RHEL6/SL6. Improved handling of HP SmartArray. Support for ext4 on RHEL5/SL5. Support for Xen paravirtualization. Integration with the local GRUB menu. Boot images can now be centralized through network transfers. Support for udev on RHEL4. Many small improvements and performance enhancements.

(more)

Release Notes: This release supports many recent distributions, including "upstart" (Ubuntu 7.10). It has more IA-64 support (RHEL5 only at the moment), better error reporting and catching, ... Debian packages (mkdeb), and improved TSM support.

(more)

[Nov 05, 2016] Affordable compact (FIT or bar form factor) 128 GB Flash drives

"FIT" form factor is the best as almost does not protrude from the USB port. There is a small (orange in case of SanDisk) indicator light and I kind of like it. When it is writing or reading, the light will blink. They are supposed to be plugged in and seldom pulled out, or pulled out fairly rarely. Perfect for local backup of OS.
Bar form factor protrude one inch or so. Which in many case is acceptable but still carry some risks.
Nov 05, 2016 | www.amazon.com

USB 3.0 FIT

USB 3.0 BAR

USB 2.0

[Nov 04, 2016] Coding Style rear-rear Wiki

Reading rear sources is an interesting exercise. It really demonstrates attempt to use "reasonable' style of shell programming and you can learn a lot.
Nov 04, 2016 | github.com

Relax-and-Recover is written in Bash (at least bash version 3 is needed), a language that can be used in many styles. We want to make it easier for everybody to understand the Relax-and-Recover code and subsequently to contribute fixes and enhancements.

Here is a collection of coding hints that should help to get a more consistent code base.

Don't be afraid to contribute to Relax-and-Recover even if your contribution does not fully match all this coding hints. Currently large parts of the Relax-and-Recover code are not yet in compliance with this coding hints. This is an ongoing step by step process. Nevertheless try to understand the idea behind this coding hints so that you know how to break them properly (i.e. "learn the rules so you know how to break them properly").

The overall idea behind this coding hints is:

Make yourself understood

Make yourself understood to enable others to fix and enhance your code properly as needed.

From this overall idea the following coding hints are derived.

For the fun of it an extreme example what coding style should be avoided:

#!/bin/bash for i in `seq 1 2 $((2*$1-1))`;do echo $((j+=i));done



   

Try to find out what that code is about - it does a useful thing.

Code must be easy to read Code should be easy to understand

Do not only tell what the code does (i.e. the implementation details) but also explain what the intent behind is (i.e. why ) to make the code maintainable.

Here the initial example so that one can understand what it is about:

#!/bin/bash # output the first N square numbers # by summing up the first N odd numbers 1 3 ... 2*N-1 # where each nth partial sum is the nth square number # see https://en.wikipedia.org/wiki/Square_number#Properties # this way it is a little bit faster for big N compared to # calculating each square number on its own via multiplication N=$1 if ! [[ $N =~ ^[0-9]+$ ]] ; then echo "Input must be non-negative integer." 1>&2 exit 1 fi square_number=0 for odd_number in $( seq 1 2 $(( 2 * N - 1 )) ) ; do (( square_number += odd_number )) && echo $square_number done

Now the intent behind is clear and now others can easily decide if that code is really the best way to do it and easily improve it if needed.

Try to care about possible errors

By default bash proceeds with the next command when something failed. Do not let your code blindly proceed in case of errors because that could make it hard to find the root cause of a failure when it errors out somewhere later at an unrelated place with a weird error message which could lead to false fixes that cure only a particular symptom but not the root cause.

Maintain Backward Compatibility

Implement adaptions and enhancements in a backward compatible way so that your changes do not cause regressions for others.

Dirty hacks welcome

When there are special issues on particular systems it is more important that the Relax-and-Recover code works than having nice looking clean code that sometimes fails. In such special cases any dirty hacks that intend to make it work everywhere are welcome. But for dirty hacks the above listed coding hints become mandatory rules:

For example a dirty hack like the following is perfectly acceptable:

# FIXME: Dirty hack to make it work # on "FUBAR Linux version 666" # where COMMAND sometimes inexplicably fails # but always works after at most 3 attempts # see http://example.org/issue12345 # Retries should have no bad effect on other systems # where the first run of COMMAND works. COMMAND || COMMAND || COMMAND || Error "COMMAND failed."

Character Encoding

Use only traditional (7-bit) ASCII charactes. In particular do not use UTF-8 encoded multi-byte characters.

Text Layout Variables Functions Relax-and-Recover functions

Use the available Relax-and-Recover functions when possible instead of re-implementing basic functionality again and again. The Relax-and-Recover functions are implemented in various lib/*-functions.sh files .

test, [, [[, (( Paired parenthesis See also

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Top articles

Sites

Contract support and subscription from developers (Gratien D'haese (IT3 Consultants)

Presentations and videos on Youtube:

linuxtag.org



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: May, 26, 2018