Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Perl and Shell Backup Scripts

News

Simple Unix Backup Tools

Recommended Books

Recommended Links

rsnapshot

Etc directory backup

Backing Up And Restoring Linux With SystemImager

Baseliners Suse supportconfig Red Hat sosreport UNIX - Linux Copy Master Boot Record (MBR) SCC Dump and restore udev
Emergency Restore Procedure Loopback filesystem Labeling the partition Booting into Rescue Mode LVM Linux backup and recovery Linux Swap filesystem
Grub How To Manage Your Disk By UUID Data deduplication Linux filesystems Sysadmin Horror Stories Humor Etc

Introduction


Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.

unknown source (well originally Paul McCartney :-)

One thing that distinguishes a professional Unix sysadmin from an amateur is attitude to backups and the level of knowledge of backup technologies. Professional sysadmin knows all too well that often the difference between major SNAFU and a nuisance is availability of the up-to-date backup of data.  Here is pretty telling poem (From unknown source (well, originally, by Paul McCartney :-) on the subject :

Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.

You can read more about this important issue at Sysadmin Horror Stories

Unix and linux suffer from neglect: there are very few good back up solutions. Dump and restore are hopelessly outdated. Tar and cpio work with individual filesystems/partitions not with the disk as whole. 

Partimage was attempt to replicate Ghost functionality on Linux, but unfortunately is no longer supported. Rsnapshot  is an interesting idea that utilizes capabiltiesi of modern USB drives and not replicated tape base "mentality" of tools like tar (used in written in bash  Relax-and-Recover) . See also Bare metal recovery of Linux systems

Poor man implementation of partimage that supports filesystems like ext4 can be created using classic Unix tools and bootable CD with the same flavor of Linux (this is an important consideration, see below). You might also need  NFS support (to mount the image)

You can create full backup of your system the following way:

In this case restore process is as following

Please not that a bootable disks like SystemRescueCD or Knoppix are not very convenient/usable to with RHEL as they do not understand security attributes used in RHEL. USB drive created from the standard RHEL ISO is the only reliable rescue disk for this flavor of Linux

The three major backup programs are dump/restore, tar, and cpio. Their functionality can be easily enhanced by Perl scripts.

Backup and restore of MBR

Create a backup of your MBR by doing a:

dd if=/dev/hdx of=MBR-backup bs=512 count=1

To restore MBR use

dd if=MBR-backup of=/dev/hdx bs=512 count=1 

the same can be done with sfdisk  utility.

When you're multi-booting or installing a new operating system onto a used system, sometimes the MBR (Master Boot Record) gets all messed up, so you need to wipe it out and start over. You can do this with the dd command. Be sure to use your own drive name for the of= value:

# dd if=/dev/zero of=/dev/hda bs=446 count=1

That preserves the partition table. If you also want to zero out the partition table, do this:

# dd if=/dev/zero of=/dev/hda bs=512 count=1

See also  discusiion at Ubuntu Blog

Michael Manning - October 20, 2005
If you are going to need to use a Live CD to recover the backup MBR then you could have booted into rescue mode from the install CD (of any linux distro you choose) and mounted the linux partition, chrooted to the root partition and typed grub-install /dev/hdx.

mkdir /mnt/temp
mount /dev/hda3 /mnt/temp ## if hda3 was the linux partition
chroot /mnt/temp
grub-install /dev/hda

That would install the grub first stage boot loader to you the MBR of the hard disk and presented you with the same options you had before the Windows reinstall. no need for backup MBR records!

Just my two bobs worth – there is always more than one way to skin a cat.
Michael Manning

Reply
 2. ubuntonista - October 20, 2005
Thanks Michael!

Reply

 3. zenith - October 20, 2005
Very useful post, like so many here. Thanks for putting together a great Ubuntu blog.

Reply

 4. Jason - October 20, 2005
Note that the grub-install method has the added protection of not inadvertantly overwriting your partition table if it had been modified since your last MBR backup.

This is because the 512byte MBR sector is actually two parts:
1) The first 446 bytes is the grub stage1 bootloader (or the windows bootloader after you’ve reinstalled windows and it “helpfully” overwrites grub).
2) The last 64 bytes is where your partition table is stored.

So, if you only want to backup the bootloader in the MBR, remember to change the bs=512 to bs=446.

Reply
 5. kOoLiNuS - October 21, 2005
this is one of the “problems” of the single disk ubuntu … they actually don’t have a real rescue mode so you need a live-cd or another distro 1st cd …

they know it, since there are some posts on this subject dated some month ago in the archives …

 

Using Perl as a glue for existing Linux backup tools

Linux does not have the traditional UNIX backup programs dump and restore. Only tar and cpio are present. It is also possible to use dump and restore in a more secure fashion over ssh.

The GNU tar supports remote devices using the same syntax as rdump.

The same could be accomplished with  by using a pipeline and rsh to send the data to a remote tape drive.

# tar cf - . | rsh hostname dd of=tape-device obs=20b

If you are worried about the security of backing up over a network you should use the ssh command instead of rsh.

cpio  can backup and restore hard links and special files. This last feature makes cpio a better choice for filesystem backup. cpio does not know how to walk the directory tree and a list of files must be provided through stdin.

cpio does not support backups across the network. You can use a pipeline and ssh to send the data to a remote server

# for f in directory_list; do
find $f >> backup.list
done
# cpio -v -o --format=newc < backup.list | ssh user@host "cat > backup_device"

Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Aug 29, 2017] backup-etc.sh -- A script to backup the /etc directory

This is simple script that generated "dot" progression lines. Backup name includes a timestamp. No rotation implemented.
Aug 29, 2017 | wpollock.com
   #!/bin/bash
# Script to backup the /etc heirarchy
#
# Written 4/2002 by Wayne Pollock, Tampa Florida USA
#
#  $Id: backup-etc,v 1.6 2004/08/25 01:42:26 wpollock Exp $
#
# $Log: backup-etc,v $
# Revision 1.6  2004/08/25 01:42:26  wpollock
# Changed backup name to include the hostname and 4 digit years.
#
# Revision 1.5  2004/01/07 18:07:33  wpollock
# Fixed dots routine to count files first, then calculate files per dot.
#
# Revision 1.4  2003/04/03 08:10:12  wpollock
# Changed how the version number is obtained, so the file
# can be checked out normally.
#
# Revision 1.3  2003/04/03 08:01:25  wpollock
# Added ultra-fancy dots function for verbose mode.
#
# Revision 1.2  2003/04/01 15:03:33  wpollock
# Eliminated the use of find, and discovered that tar was working
# as intended all along!  (Each directory that find found was
# recursively backed-up, so for example /etc, then /etc/mail,
# caused /etc/mail/sendmail.mc to be backuped three times.)
#
# Revision 1.1  2003/03/23 18:57:29  wpollock
# Modified by Wayne Pollock:
#
# Discovered not all files were being backed up, so
# added "-print0 --force-local" to find and "--null -T -"
# to tar (eliminating xargs), to fix the problem when filenames
# contain metacharacters such as whitespace.
# Although this now seems to work, the current version of tar
# seems to have a bug causing it to backup every file two or
# three times when using these options!  This is still better
# than not backing up some files at all.)
#
# Changed the logger level from "warning" to "error".
#
# Added '-v, --verbose' options to display dots every 60 files,
# just to give feedback to a user.
#
# Added '-V, --version' and '-h, --help' options.
#
# Removed the lock file mechanism and backup file renaming
# (from foo to foo.1), in favor of just including a time-stamp
# of the form "yymmdd-hhmm" to the filename.
#
#

PATH=/bin:/usr/bin

# The backups should probably be stored in /var somplace:
REPOSITORY=/root
TIMESTAMP=$(date '+%Y%m%d-%H%M')
HOSTNAME=$(hostname)
FILE="$REPOSITORY/$HOSTNAME-etc-full-backup-$TIMESTAMP.tgz"

ERRMSGS=/tmp/backup-etc.$$
PROG=${0##*/}
VERSION=$(echo $Revision: 1.6 $ |awk '{print$2}')
VERBOSE=off

usage()
{  echo "This script creates a full backup of /etc via tar in $REPOSITORY."
   echo "Usage: $PROG [OPTIONS]"
   echo '  Options:'
   echo '    -v, --verbose   displays some feedback (dots) during backup'
   echo '    -h, --help      displays this message'
   echo '    -V, --version   display program version and author info'
   echo
}

dots()
{  MAX_DOTS=50
   NUM_FILES=`find /etc|wc -l`
   let 'FILES_PER_DOT = NUM_FILES / MAX_DOTS'
   bold=`tput smso`
   norm=`tput rmso`
   tput sc
   tput civis
   echo -n "$bold(00%)$norm"
   while read; do
      let "cnt = (cnt + 1) % FILES_PER_DOT"
      if [ "$cnt" -eq 0 ]
      then
         let '++num_dots'
         let 'percent = (100 * num_dots) / MAX_DOTS'
         [ "$percent" -gt "100" ] && percent=100
         tput rc
         printf "$bold(%02d%%)$norm" "$percent"
         tput smir
         echo -n "."
         tput rmir
      fi
   done
   tput cnorm
   echo
}

# Command line argument processing:
while [ $# -gt 0 ]
do
   case "$1" in
      -v|--verbose)  VERBOSE=on; ;;
      -h|--help)     usage; exit 0; ;;
      -V|--version)  echo -n "$PROG version $VERSION "
                     echo 'Written by Wayne Pollock '
                     exit 0; ;;
      *)             usage; exit 1; ;;
   esac
   shift
done

trap "rm -f $ERRMSGS" EXIT

cd /etc

# create backup, saving any error messages:
if [ "$VERBOSE" != "on" ]
then
    tar -cz --force-local -f $FILE . 2> $ERRMSGS 
else
    tar -czv --force-local -f $FILE . 2> $ERRMSGS | dots
fi

# Log any error messages produced:
if [ -s "$ERRMSGS" ]
then logger -p user.error -t $PROG "$(cat $ERRMSGS)"
else logger -t $PROG "Completed full backup of /etc"
fi

exit 0

Perl-based Backup Manager download

SourceForge.net

Command-line perl script with configuration files that allow you to mix and match backup tools to manage your monthly backups. You can for eg, configure a command to download files, zip them and copy it into another disk, all using different tools.

Perl-based Backup Manager Web Site

BackupPC

is a high-performance, enterprise-grade backup system for backing up Linux, Win32, and laptops to a server's disk. Features include clever pooling of identical files, no client-side software, and a powerful Apache/CGI user interface. It is written in Perl

ESR Backup

ESR Backup (Easy, Secure, Remote) is a backup system for UNIX and Linux that is easy to use and configure. It automatically performs full and incremental backups on files you choose, and automatically rotates and archives your backups, encrypts archives with a 448 bit key, and can store archives on a remote server via FTP or on the local file system.

Simplebackup

Simplebackup is a cross-platform backup program. It reads a configuration file, then it builds a compressed file for each of your backup directories or files on your backup list, and places the compressed files into another location. For example, this location can be a network mapped drive in Windows, an NFS mounted drive in Unix, another hard disk, an FTP server, an SFTP (Secure FTP) server, an HTTP (WebDAV) server, one or more email accounts, or a tape device (Unix only). This will duplicate your information, doing the so called "backup".

storebackup

Written in Perl

freshmeat.net

storebackup is a backup utility that stores files on other disks. It includes several optimizations that reduce the disk space needed and improve performance, and unifies the advantages of traditional full and incremental backups. Depending on its contents, every file is stored only once on disk. It includes tools for analyzing backup data and restoring. Once archived, files are accessible by mounting filesystems (locally, or via Samba or NFS). It is easy to install and configure.

Backup Manager

Backup Manager is a tool for generating archives easily. It is designed for those who don't want an obfuscated tool for backing up their system. It can make tar, tar.gz, tar.bz2, and zip archives, can perform incremental backups, and can upload archives to remote hosts with FTP, SSH, or RSYNC. It can be run in a parallel mode with different configuration files. The configuration file is simple to understand. The backup process can also be customized thanks to hook scripts.

flexbackup

flexbackup is a configurable and easy to use Perl-based backup tool, that can backup local files as well as remote machines (using ssh). It allows the backup itself to be made with afio, cpio, tar, dump, star, or pax. It can work with tape drives, on-disk archive files, or on-disk directory trees.

dobackup.pl

dobackup.pl is a flexible Perl script to handle unattended incremental backups of multiple servers. It handles multiple media sets with automatic media preparation and rotation, configurable 'what-to-backup', global per-host exclusion patterns, and user settable 'don't-back-this-up' metafiles. Its design goal is zero-maintenance, nothing to do except change the media when told.

fsbackup by Maxim Chirkov

Perl-based file system backup and synchronization utility.

fsbackup is an incremental backup creation utility. It supports backup compression and encryption. Backups can be stored on the local file system, and a remote host (via SSH, or FTP). Some additional scripts allow backup SQL tables from PostgreSQL and MySQL, save system configuration files, and a list of installed packages. Backed-up files can be recovered, and system packages can be reinstalled.

confstore

A configuration backup utility that makes backups of configuration.

Confstore is a configuration backup utility. It scans a system for all recognised configuration files and then stores them in a simple archive. It knows what to scan for by reading a definitions file. Confstore can also restore configuration from backup archives it has previously created.

NasBackup

NasBackup is a solution for backing up desktop PCs, laptops, and servers to network disks. It is highly configurable and uses rsync to only transfer file differences over the network.

Shell-based

backup2l

A lightweight tar-based backup/restore tool.

Bontmia

An incremental network backup tool for snapshotting directories.

Base Utilities

Partition Image is a Linux/UNIX utility similar to Symantec's Ghost. This utility saves partitions in the EXT2, Reiserfs, NTFS, HPFS, FAT16, and FAT32 file system formats to an image file. The image file can be compressed with gzip or bzip2 in order to save disk space, and it can be split in order to fit onto a series of floppy disks. This program can be useful for backup purposes. A boot/root disk is also provided, allowing you to run Partition Image without Linux installed on the hard disk.

Dump-Restore

The dump package contains both dump and restore. Dump examines files in a filesystem, determines which ones need to be backed up, and copies those files to a specified disk, tape or other storage medium. The restore command performs the inverse function of dump; it can restore a full backup of a filesystem. Subsequent incremental backups can then be layered on top of the full backup. Single files and directory subtrees may also be restored from full or partial backups.


Top Visited
Switchboard
Latest
Past week
Past month