ddrescue data recovery

This GNU manual is for  ddrescue (version 1.14, 10 January 2011).

• Introduction: Purpose and features of GNU ddrescue
• Basic concepts: Blocks, clusters, devices, files, sectors, etc
• Algorithm: How ddrescue recovers the data
• Invoking ddrescue: Command line interface
• Logfile Structure: Detailed format of the logfile
• Examples: A small tutorial with examples
• Direct Disc Access: Bypassing the kernel cache
• Fill Mode: Selectively overwriting the output file
• Generate Logfile Mode: Generating an approximate logfile

Introduction

GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying hard to rescue data in case of read errors.  The basic operation of ddrescue is fully automatic. That is, you don't have to wait for an error, stop the program, read the log, run it in reverse mode, etc.

If you use the logfile feature of ddrescue, the data is rescued very efficiently, (only the needed blocks are read). Also you can interrupt the rescue at any time and resume it later at the same point.

Ddrescue does not write zeros to the output when it finds bad sectors in the input, and does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps without wiping out the data already rescued.  That gives you possibility to try to read CDs on different devices.

Automatic merging of backups: If you have two or more damaged copies of a file, cdrom, etc, and run ddrescue on all of them, one at a time, with the same output file, you will probably obtain a complete and error-free file. This is so because the probability of having damaged areas at the same places on different input files is very low. Using the logfile, only the needed blocks are read from the second and successive copies.

Ddrescue recommends lzip for compression of backups because of its reliability and data recovery capabilities, including error-checked merging of backup copies. The combination ddrescue + lziprecover is the best option for recovering data from multiple damaged copies. See lzip-example, for an example

Recordable CD and DVD media keep their data only for a finite time (typically for many years). After that time, data loss develops slowly with read errors growing from the outer media region towards the inside. Just make two (or more) copies of every important CD/DVD you burn so that you can later recover them with ddrescue.

Because ddrescue needs to read and write at random places, it only works on seekable (random access) input and output files.

If your system supports it, ddrescue can use direct disc access to read the input file, bypassing the kernel cache.

Ddrescue also features a "fill mode" able to selectively overwrite parts of the output file, which has a number of interesting uses like wiping data, marking bad areas or even, in some cases, "repair" damaged sectors.

 Basic concepts

• Block Any amount of data. A block is described by its starting position and its size.
   
• Cluster Group of consecutive sectors read or written in one go.
   
• Device Piece of hardware containing data. Hard disc drives, cdrom drives, USB pendrives, are devices. /dev/hda, /dev/sdb, are device names.
   
• File Files are named units of data which are stored by the operating system for you to retrieve later by name. Devices are accessed by means of their associated file names.
   
• Partition Every part in which a device is divided. A partition normally contains a filesystem. /dev/hda1, /dev/sdb3, are partition names.
   
• Recoverable formats As ddrescue uses standard library functions to read data from the device being rescued, only mountable device formats can be rescued with ddrescue. DVDs can be mounted and they can be rescued, "compact disc digital audio" CDs can't, "video CDs"[1] maybe.
  [1] http://en.wikipedia.org/wiki/Video_CD
   
• Sector Hardware block. Smallest accessible amount of data on a device.

Algorithm

GNU ddrescue manages efficiently the status of the rescue in progress and tries to rescue the good parts first, maximizing the amount of data finally recovered from a failing drive.

The standard dd utility can be used to save data from a failing drive, but it reads the data secuentially, which may wear out the drive without rescuing anything if the errors are at the beginning of the drive.

Other programs switch to small size reads when they find errors, but this is a bad idea because it means spending more time at error areas, damaging the surface, the heads and the drive mechanics, instead of getting out of them as fast as possible. This behavior reduces the chances of rescuing the remaining good data.

The algorithm of ddrescue is as follows (the user may interrupt the process at any point, but be aware that a bad drive can block ddrescue for a long time until the kernel gives up):

1) Optionally read a logfile describing the status of a multi-part or previously interrupted rescue. If no logfile is specified or is empty or does not exist, mark all the rescue domain as non-tried.

2) Read the non-tried parts of the input file, marking the failed blocks as non-trimmed and skipping beyond them, until all the rescue domain is tried. Only non-tried areas are read in large blocks. Trimming, splitting and retrying are done sector by sector. Each sector is tried at most two times; the first in this step as part of a large block read, the second in one of the steps below as a single sector read.

3) Read backwards one sector at a time the non-trimmed blocks, until a bad sector is found. For each non-trimmed block, mark the bad sector found as bad-sector and mark the rest of that block as non-split.

4) Read forwards one sector at a time the non-split blocks, marking the bad sectors found as bad-sector. After a number of consecutive bad sectors is found in a block large enough, the block is split by half and the reading continues on the second half. This recursively splits the largest failed blocks without producing a logfile too large.

5) Optionally try to read again the bad sectors until the specified number of retries is reached.

6) Optionally write a logfile for later use. 

The logfile is periodically saved to disc, as well as when ddrescue finishes or is interrupted. So in case of a crash you can resume the rescue with little recopying.

Also, the same logfile can be used for multiple commands that copy different areas of the input file, and for multiple recovery attempts over different subsets. See this example:

Rescue the most important part of the disc first.

     ddrescue -i0 -s50M /dev/hdc hdimage logfile
     ddrescue -i0 -s1M -d -r3 /dev/hdc hdimage logfile

Then rescue some key disc areas.

     ddrescue -i30G -s10G /dev/hdc hdimage logfile
     ddrescue -i230G -s5G /dev/hdc hdimage logfile

Now rescue the rest (does not recopy what is already done).

     ddrescue /dev/hdc hdimage logfile
     ddrescue -d -r3 /dev/hdc hdimage logfile

Invoking ddrescue

The format for running ddrescue is:

     ddrescue [options] infile outfile [logfile]

ddrescue supports the following options:

`-h'

`--help'

Print an informative help message describing the options and exit.
 

`-V'

`--version'

Print the version number of ddrescue on the standard output and exit.
 

`-b bytes'

`--block-size=bytes'

Sector (hardware block) size of input device in bytes (usually 512 for hard discs and 3.5" floppies, 1024 for 5.25" floppies, and 2048 for cdroms). Defaults to 512.
 

`-B'

`--binary-prefixes'

Show units with binary prefixes (powers of 1024).
SI prefixes (powers of 1000) are used by default. (See table below).
 

`-c sectors'

`--cluster-size=sectors'

Number of sectors to copy at a time. Defaults to 64KiB / sector_size. Try smaller values for slow drives. The number of sectors per track (18 or 9) is a good value for floppies.
 

`-C'

`--complete-only'

Limit rescue domain to the blocks listed in the logfile. Do not read new data beyond logfile limits. This is useful when reading from devices of undefined size, like raw devices.
 

`-d'

`--direct'

Use direct disc access to read the input file, bypassing the kernel cache. (Open the file with the O_DIRECT flag). Use it only on devices or partitions, not on regular files. Sector size must be correctly set for this to work. Not all systems support this.

If your system does not support direct disc access, ddrescue will warn you. If the sector size is not correctly set, all reads will result in errors, and no data will be rescued.
 

`-D'

`--synchronous'

Use synchronous writes for output file. (Issue a fsync call after every write). May be useful when forcing the drive to remap its bad sectors.
 

`-e [+]n'

`--max-errors=[+]n'

Maximum number of error areas allowed before giving up. Defaults to infinity. If n is preceded by `+' the number refers to new error areas found in this run, not counting those already annotated in the logfile.
 

`-E bytes'

`--max-error-rate=bytes'

Maximum growth per second of error size, in bytes, allowed before giving up. Defaults to infinity.
 

`-f'

`--force'

Force overwrite of output file. Needed when the output file is not a regular file, but a device or partition.
 

`-F types'

`--fill=types'

Fill the output file blocks specified as any of types in the logfile, with data read from the input file. types contains one or more of the status characters defined in the chapter Logfile Structure (see Logfile Structure). See the chapter Fill Mode (see Fill Mode) for a complete description of the fill mode.
 

`-g'

`--generate-logfile'

Generate an approximate logfile from the input and output files of the original rescue run. Note that you must keep the original offset between `--input-position' and `--output-position' of the original rescue run.
 

`-i pos'

`--input-position=pos'

Starting position in input file, in bytes. Defaults to 0. In fill mode it refers to the original input file. See the chapter Fill Mode (see Fill Mode) for details.
 

`-m file'

`--domain-logfile=file'

Restrict the rescue domain to the blocks marked as finished in the logfile file. This is useful if the destination drive fails during the rescue.
 

`-M'

`--retrim'

Mark all failed blocks inside the rescue domain as non-trimmed before beginning the rescue. The effect is similar to `--max-retries=1', but the bad sectors are tried in a different order, making perhaps possible to rescue some of them.
 

`-n'

`--no-split'

Stop after the trimming pass. Avoids spending a lot of time trying to rescue the most difficult parts of the file. This option overrides the `--max-retries' option.
 

`-o pos'

`--output-position=pos'

Starting position in output file, in bytes. Defaults to `--input-position'. The bytes below pos aren't touched if they exist and truncation is not requested. Else they are set to 0.
 

`-p'

`--preallocate'

Preallocate space on disc for output file. Only space for regular files can be preallocated. If preallocation succeeds, rescue will not fail due to lack of free space on disc. If ddrescue can't determine the size to preallocate, you may need to specify it with some combination of the `--input-position', `--output-position', `--max-size', and `--domain-logfile' options.
 

`-q'

`--quiet'

Quiet operation. Suppress all messages.
 

`-r n'

`--max-retries=n'

Exit after given number of retry passes. Defaults to 0. -1 means infinity. Every bad sector is tried only one time per pass. To retry bad sectors detected on a previous run, you must specify a non-zero number of retries.
 

`-R'

`--reverse'

Reverse direction of all copy operations. This means the copy of the non-tried blocks, splitting and retrying are run backwards, while trimming is run forwards.
 

`-s bytes'

`--max-size=bytes'

Maximum size of the input data to be copied, in bytes. If ddrescue can't determine the size of the input device, you may need to specify it with this option. Note that this option specifies the size of the input data to be copied, not the size of the output file. So, for example, the following command creates an output file 300 bytes long, but only writes data on the last 200 bytes:

          ddrescue -i 100 -s 200 infile outfile logfile
     

 

`-S'

`--sparse'

Use sparse writes for output file. (The blocks of zeros are not actually allocated on disc). May save a lot of disc space in some cases. Not all systems support this.
 

`-t'

`--truncate'

Truncate output file to zero size before writing to it. Only works for regular files, not for drives or partitions.
 

`-T'

`--try-again'

Mark all non-split and non-trimmed blocks inside the rescue domain as non-tried before beginning the rescue. Try this if the drive stops responding and ddrescue immediately starts splitting failed blocks when restarted. If `--retrim' is also specified, mark all failed blocks inside the rescue domain as non-tried.
 

`-v'

`--verbose'

Verbose mode.

Table of SI and binary prefixes (unit multipliers):

Logfile Structure

The logfile is a text file easy to read and edit. It is formed by three parts, the heading comments, the status line, and the list of data blocks.

Any line beginning with `#' is a comment line. The blocks in the list of data blocks must be contiguous and non-overlapping.

NOTE: Logfiles generated by a version of ddrescue prior to 1.6 lack the status line. If you want to use an old logfile with ddrescue 1.6 or later, you will have to insert a line like `0 +' at the beginning of the logfile.

The heading comments contain the version of ddrescue and the command line used to create the logfile. They are intended as information for the user.

The first non-comment line is the status line. It contains a non-negative integer and a status character. The integer is the position being tried in the input file. The status character is one of these:

Every line in the list of data blocks describes a block of data. It contains 2 non-negative integers and a status character. The first integer is the starting position of the block in the input file, the second integer is the size (in bytes) of the block. The status character is one of these:

And here is an example logfile:

# Rescue Logfile. Created by GNU ddrescue version 1.14
# Command line: ddrescue /dev/fd0 fdimage logfile
# current_pos current_status
0x00120000 ?
# pos size status
 

If you edit the file, you may use decimal, hexadecimal or octal values, using the same syntax that integer constants in C++.

Examples

Ddrescue is like any other power tool. You need to understand what it does, and you need to understand some things about the machines it does those things to, in order to use it safely.

A failing drive tends to develop more and more errors as time passes. Because of this, you should rescue the data from a drive as soon as you notice the first error. Be diligent because every time a physicaly damaged drive powers up and is able to output some data, it may be the very last time that it ever will.

You should make a copy of the failing drive with ddrescue, and then try to repair the copy. If your data is really important, use the first copy as a master for a second copy, and try to repair the second copy. If something goes wrong, you have the master intact to try again.

IMPORTANT! Always use a logfile unless you know you won't need it. Without a logfile, ddrescue can't resume a rescue, only reinitiate it.

IMPORTANT! Never try to rescue a r/w mounted partition. The resulting copy may be useless.

IMPORTANT! If you use a device or a partition as destination, any data stored there will be overwritten.

IMPORTANT! If you interrupt the rescue and then reboot, any partially copied partitions should be hidden before allowing them to be touched by any operating system that tries to mount and "fix" the partitions it sees.

IMPORTANT! Never try to repair a file system on a drive with I/O errors; you will probably lose even more data.

If you are trying to rescue a whole partition, first repair the copy with e2fsck or some other tool appropiate for the type of partition you are trying to rescue, then mount the repaired copy somewhere and try to recover the files in it.

If the drive is so damaged that the file system in the rescued partition can't be repaired or mounted, you will have to browse the rescued data with an hex editor and extract the desired parts by hand or use a file recovery tool like photorec.

If the partition table is damaged, you may try to rescue the whole disc, then try to repair the partition table and the partitions on the copy.

If the damaged drive is not listed in /dev, then you cannot rescue it. At least not with ddrescue.

Example 1: Rescue a whole disc with two ext2 partitions in /dev/hda to /dev/hdb.

Note: you do not need to partition /dev/hdb beforehand, but if the partition 
table on /dev/hda is damaged, you'll need to recreate it somehow on /dev/hdb. 

     ddrescue -f -n /dev/hda /dev/hdb logfile
     ddrescue -d -f -r3 /dev/hda /dev/hdb logfile
     fdisk /dev/hdb
     e2fsck -v -f /dev/hdb1
     e2fsck -v -f /dev/hdb2

     ddrescue -f -n /dev/hda2 /dev/hdb2 logfile
     ddrescue -d -f -r3 /dev/hda2 /dev/hdb2 logfile
     e2fsck -v -f /dev/hdb2
     mount -t ext2 -o ro /dev/hdb2 /mnt
       (read rescued files from /mnt)

     ddrescue -n -b2048 /dev/cdrom cdimage logfile
     ddrescue -d -b2048 /dev/cdrom cdimage logfile
       (write cdimage to a blank CD-ROM)

     ddrescue -n -b2048 /dev/cdrom cdimage logfile
     ddrescue -d -b2048 /dev/cdrom cdimage logfile
       (insert second copy in the CD drive)
     ddrescue -d -r1 -b2048 /dev/cdrom cdimage logfile
       (write cdimage to a blank CD-ROM)

     ddrescue -b2048 /dev/cdrom cdimage1 logfile1
     mount -t iso9660 -o loop,ro cdimage1 /mnt/cdimage
     cp /mnt/cdimage/backup.tar.lz rescued1.tar.lz
     umount /mnt/cdimage
       (insert second copy in the CD drive)
     ddrescue -b2048 /dev/cdrom cdimage2 logfile2
     mount -t iso9660 -o loop,ro cdimage2 /mnt/cdimage
     cp /mnt/cdimage/backup.tar.lz rescued2.tar.lz
     umount /mnt/cdimage
     lziprecover --merge -o rescued.tar.lz rescued1.tar.lz rescued2.tar.lz

     ddrescue -f -n /dev/hda /dev/hdb logfile1     <-- /dev/hdb fails here
     ddrescue -f -m logfile1 /dev/hdb /dev/hdc logfile2
     ddrescue -f -n /dev/hda /dev/hdc logfile2
     ddrescue -d -f -r3 /dev/hda /dev/hdc logfile2

     ddrescue -f -n /dev/hda /dev/hdb logfile      <-- /dev/hda fails here
       (restart /dev/hda or reboot computer as many times as needed)
     ddrescue -f -n -T /dev/hda /dev/hdb logfile
     ddrescue -d -f -r3 /dev/hda /dev/hdb logfile

Direct Disc Access

If you notice that the sizes and offsets in the log file are ALWAYS multiples of the sector size, maybe your kernel is caching the disc accesses and grouping them. In this case you may want to use direct disc access or a raw device to bypass the kernel cache and rescue more of your data.

NOTE! Sector size must be correctly set with the `--block-size' option for this to work. Try the `--direct' option first. If direct disc access is not available in your system, try raw devices. Read your system documentation to find how to bind a raw device to a regular block device.

Ddrescue aligns its I/O buffer to the sector size so that it can be used for direct disc access or to read from raw devices. For efficiency reasons, also aligns it to the memory page size if page size is a multiple of sector size. Ddrescue can't determine the size of a raw device, so a explicit `--max-size' or `--complete-only' option is needed.

Using direct disc access, or reading from a raw device, may be slower or faster than normal cached reading depending on your OS and hardware. In case it is slower you may want to make a first pass using normal cached reads and use direct disc access, or a raw device, only to recover the good sectors inside the failed blocks.

     ddrescue -f -n /dev/hdb1 /dev/hdc1 logfile
     ddrescue -d -f -r3 /dev/hdb1 /dev/hdc1 logfile
     e2fsck -v -f /dev/hdc1
     mount -t ext2 -o ro /dev/hdc1 /mnt

     raw /dev/raw/raw1 /dev/hdb1
     ddrescue -f -n /dev/hdb1 /dev/hdc1 logfile
     ddrescue -C -f -r3 /dev/raw/raw1 /dev/hdc1 logfile
     raw /dev/raw/raw1 0 0
     e2fsck -v -f /dev/hdc1
     mount -t ext2 -o ro /dev/hdc1 /mnt

Fill Mode

When ddrescue is invoked with the `--fill' option it operates in "fill mode", which is different from the default "rescue mode". That is, if you use the `--fill' option, ddrescue does not rescue anything. It only fills with data read from the input file the blocks of the output file whose status character from the logfile coincides with one of the type characters specified as argument to the `--fill' option.

In fill mode the input file may have any size. If it is too small, the data will be duplicated as many times as necessary to fill the input buffer. If it is too big, only the needed data will be read.

Note that in fill mode the input file is always read from position 0. If you specify a `--input-position', it refers to the original input file from which the logfile was built, and is only used to calculate the offset between input and output positions.

Note also that when filling the input file of the original rescue run you should set `--input-position' and `--output-position' to identical values, whereas when filling the output file of the original rescue run you should keep the original offset between `--input-position' and `--output-position'.

The `--fill' option implies the `--complete-only' option.

In fill mode the logfile is updated to allow resumability when interrupted or in case of a crash, but as nothing is being rescued the logfile is not destroyed. The status line is the only part of the logfile that is modified.

Example 1: Mark parts of the rescued copy to allow finding them when examined in an hex editor. For example, the following command line fills all blocks marked as `-' (bad-sector) with copies of the string `BAD SECTOR ':

     echo -n "BAD SECTOR " > tmpfile
     ddrescue --fill=- tmpfile outfile logfile

Example 2: Wipe only the good sectors, leaving the bad sectors alone. This way, the drive will still test bad (i.e., with unreadable sectors). This is the fastest way of wiping a failing drive, and is specially useful when sending the drive back to the manufacturer for warranty replacement.

     ddrescue --fill=+ --force /dev/zero bad_drive logfile

Example 3: Force the drive to remap the bad sectors, making it usable again. If the drive has only a few bad sectors, and they are not caused by drive age, you can probably just rewrite those sectors, and the drive will reallocate them automatically to new "spare" sectors that it keeps for just this purpose. WARNING! This may not work on your drive.

     ddrescue --fill=- --force --synchronous /dev/zero bad_drive logfile

1) Copy the damaged drive with ddrescue until finished. Do not use sparse writes. This yields a logfile with only finished (`+') and bad-sector (`-') blocks.

2) Mount the copied drive (or the image file, via loopback device).

3) Compute a md5sum or other checksum for every file. Build a list of all the files and their checksums.

4) Fill the bad-sector blocks of the copied drive or image file with a byte value different from zero.

5) Verify the checksums. Those files which have different checksums this time reside (at least partially) in damaged disc areas.

6) Optionally fill the bad-sector blocks of the copied drive or image file with zeros to restore the disc image.

Generate-logfile Mode

So you didn't read the tutorial and started ddrescue without a logfile. Now, two days later, your computer crashed and you can't know how much data ddrescue managed to save. And even worse, you can't resume the rescue; you have to restart it from the very beginning.

Or maybe you started copying a drive with `dd conv=noerror,sync' and are now in the same situation described above. In this case, note that you can't use a copy made by dd unless it was invoked with the `sync' conversion argument.

Don't despair (yet). Ddrescue can in some cases generate an approximate logfile, from the input file and the (partial) copy, that is almost as good as an exact logfile. It makes this by simply assuming that sectors containing all zeros were not rescued.

However, if the destination of the copy was a drive or a partition, (or an existing regular file and truncation was not requested), most probably you will need to restart ddrescue from the very beginning. (This time with a logfile, of course). The reason is that old data may be present in the drive that have not been overwritten yet, and may be thus non-tried but non-zero.

For example, if you first tried one of these commands:

     ddrescue infile outfile
     or
     dd if=infile of=outfile conv=noerror,sync

you can generate an approximate logfile with this command:

     ddrescue --generate-logfile infile outfile logfile

NEWS CONTENTS

• 20161226 : How to Check Bad Sectors or Bad Blocks on Hard Disk in Linux by Aaron Kili ( How to Check Bad Sectors or Bad Blocks on Hard Disk in Linux, Dec 26, 2016 )
• 20111023 : Weekend Project Rescue Failing Drives With SystemRescue ( Linux.com )
• 20100708 : Save Your Data from the Click of Death by Kyle Rankin ( July 14, 2004 , O'Reilly Sysadmin )
• 20100311 : g4u ( g4u, Mar 11, 2010 )
• 20100311 : Creating a dd-dcfldd Image Using Automated Image & Restore (AIR) HowtoForge - Linux Howtos and Tutorials ( Creating a dd-dcfldd Image Using Automated Image & Restore (AIR) HowtoForge - Linux Howtos and Tutorials, )
• 20100311 : recover-data-from-a-dead-hard-drive-using-ddrescue ( recover-data-from-a-dead-hard-drive-using-ddrescue, )
• 20070215 : Recover Data from a Damaged hard disk using dd_rhelp ( Debian Admin )

Old News ;-)

[Dec 26, 2016] How to Check Bad Sectors or Bad Blocks on Hard Disk in Linux by Aaron Kili

October 26, 2016 |

Let us start by defining a bad sector/block, it's a section on a disk drive or flash memory that can not be read from or written to anymore, as a result of a fixed physical damage on the disk surface or failed flash memory transistors.

As bad sectors continue to accumulate, they can undesirably or destructively affect your disk drive or flash memory capacity or even lead to a possible hardware failure.

It is also important to note that the presence of bad blocks should alert you to start thinking of getting a new disk drive or simply mark the bad blocks as unusable.

Therefore, in this article, we will go through the necessary steps that can enable you determine the presence or absence of bad sectors on your Linux disk drive or flash memory using certain disk scanning utilities.

That said, below are the methods:

Check Bad Sectors in Linux Disks Using badblocks Tool

A badblocks program enables users to scan a device for bad sectors or blocks. The device can be a hard disk or an external disk drive, represented by a file such as /dev/sdc.

Firstly, use the fdisk command with superuser privileges to display information about all your disk drives or flash memory plus their partitions:

$ sudo fdisk -l

List Linux Filesystem Partitions

Then scan your Linux disk drive to check for bad sectors/blocks by typing:

$ sudo badblocks -v /dev/sda10 > badsectors.txt

Scan Hard Disk Bad Sectors in Linux

In the command above, badblocks is scanning device /dev/sda10 (remember to specify your actual device) with the -v enabling it to display details of the operation. In addition, the results of the operation are stored in the file badsectors.txt by means of output redirection.

In case you discover any bad sectors on your disk drive, unmount the disk and instruct the operating system not to write to the reported sectors as follows.

You will need to employ e2fsck (for ext2/ext3/ext4 file systems) or fsck command with the badsectors.txt file and the device file as in the command below.

The -l option tells the command to add the block numbers listed in the file specified by filename (badsectors.txt) to the list of bad blocks.

------------ Specifically for ext2/ext3/ext4 file-systems ------------ 
$ sudo e2fsck -l badsectors.txt /dev/sda10
OR
------------ For other file-systems ------------ 
$ sudo fsck -l badsectors.txt /dev/sda10

Scan Bad Sectors on Linux Disk Using Smartmontools

This method is more reliable and efficient for modern disks (ATA/SATA and SCSI/SAS hard drives and solid-state drives) which ship in with a S.M.A.R.T (Self-Monitoring, Analysis and Reporting Technology) system that helps detect, report and possibly log their health status, so that you can figure out any impending hardware failures.

You can install smartmontools by running the command below:

------------ On Debian/Ubuntu based systems ------------ 
$ sudo apt-get install smartmontools
------------ On RHEL/CentOS based systems ------------ 
$ sudo yum install smartmontools

Once the installation is complete, use smartctl which controls the S.M.A.R.T system integrated into a disk. You can look through its man page or help page as follows:

$ man smartctl
$ smartctl -h

Now execute the smartctrl command and name your specific device as an argument as in the following command, the flag -H or --health is included to display the SMART overall health self-assessment test result.

$ sudo smartctl -H /dev/sda10

Check Linux Hard Disk Health

The result above indicates that your hard disk is healthy, and may not experience hardware failures any soon.

For an overview of disk information, use the -a or --all option to print out all SMART information concerning a disk and -x or --xall which displays all SMART and non-SMART information about a disk.

In this tutorial, we covered a very important topic concerning disk drive health diagnostics, you can reach us via the feedback section below to share your thoughts or ask any questions and remember to always stay connected to Tecmint.

[Oct 23, 2011] Weekend Project Rescue Failing Drives With SystemRescue

Linux.com

When a hard drive, CD/DVD, USB stick, or any digital storage media is on its way to the Great Bitbucket in the Sky, GNU ddrescue is my favorite data recovery tool. GNU ddrescue is included in the default SystemRescue image. Before we dive into the fun stuff, there is some vexing naming confusion to clear up. There are two ddrescue programs in SystemRescue. GNU ddrescue, by Antonio Diaz, is the one I prefer. The version on the current SystemRescue release is ddrescue 1.14. There is also a dd_rescue, version 1.23, by Kurt Garloff. dd_rescue is nice, but it's slower than ddrescue and doesn't include as many features.

Just to keep it interesting, Debian Linux adds its own bizarre naming conventions. The Debian package name for GNU ddrescue is gddrescue, and the package name for dd_rescue is ddrescue. But the binary for gddrescue is /sbin/ddrescue, and the binary for dd_rescue is /bin/dd_rescue. Fortunately, SystemRescue doesn't mess with the original binary names, and calls them /usr/bin/ddrescue and /bin/dd_rescue.

Enough of that; let's talk about what makes GNU ddrescue my favorite. It performs block-level copies of the failing media, and so it doesn't matter what filesystem is on the media. You're probably thinking it sounds like the venerable dd command, and it is similar, with some significant improvements. dd works fine on healthy disks, but when it encounters a read error it stops, and you have to manually restart it. It reads the media sequentially, which is very slow, and if there are a lot of bad blocks it may never complete a full pass.

GNU ddrescue is fully automatic and fast for a block-level copy program, and you want speed when a drive full of important data is dying. It seeks out good blocks to copy and skips over the bad blocks. It optionally records all activity in a logfile, so you can resume where you left off if the copying is interrupted for any reason. It is best to always generate a logfile, because every time you power up the failing drive the more likely it is to die completely. Using a logfile ensures that ddrescue will not repeat operations, but will move on and look for new good blocks to copy.

When you are rescuing a failing drive, the first step is to copy it with ddrescue. Then take the original offline, and perform any additional recovery operations on the copy. Don't touch the original any more than you have to. You can copy the copy as many times as you need for insurance.

You need a healthy drive to copy your rescued data to. I prefer USB-attached media such as a USB hard drive, USB thumb drive, Compact Flash, or SD cards. Of course a second internal hard drive is a good option, or this might be your chance to finally use that eSATA port that always looked like it should be cool and useful, but you never found a reason to use it. Your second drive should be at least 50% larger than the drive you're recovering. The troubled drive must not be mounted. The simplest invocation looks like this:

# ddrescue /dev/sda1 /dev/sdb1 logfile

Here, /dev/sda1 is a partition on the failing drive. Everything on /dev/sdb1 will be overwritten, and the logfile will be written to /dev/sdb1. You can name the logfile anything you want. You can rescue an entire drive if you prefer, like this:

# ddrescue /dev/sda /dev/sdb logfile

Note that if there is more than one partition on the failing drive and the partition table is damaged, you will have to re-create it on the rescue drive. I copy one partition at a time to avoid this sort of drama.

You can have ddrescue make multiple passes with the -r option; sometimes you can make a more complete recovery this way. You can go as high as you want; I use 3-5:

# ddrescue -r5 /dev/sda2 /dev/sdb1 logfile

Sometimes ddrescue is nearly magical for rescuing scratched CDs and DVDs. The first command copies the disk, and the second command copies it to a blank disk:

# ddrescue -n -b2048 /media/cdrom image logfile
# ddrescue -d -b2048 /media/cdrom image logfile

You can give the image file whatever name you like. While I've never needed to go beyond the basics in this article, ddrescue has a whole lot of other capabilities that you can learn about in the GNU ddrescue manual.

[Jul 08, 2010] Save Your Data from the Click of Death by Kyle Rankin

July 14, 2004 | O'Reilly Sysadmin

Related link: http://www.garloff.de/kurt/linux/ddrescue/

One thing I like about Linux is that there are always tools out there to discover that save you a lot of time or in this case data. I always love to walk a new user through transferring files from one drive to another and tell them "now run watch df to check the output."

"Watch? Oh WOW why didn't I know about this before?"

Well, I had another one of those moments the other day when I ran across dd_rescue. This utility addresses the issue of using dd to image a drive that has bad blocks or otherwise has errors. When dd hits an error, it fails, so you are left with using skip and seek arguments to try to route around the bad blocks. Dd_rescue addresses this by simply skipping bad blocks it finds (or trying X number of times before skipping if you wish), so you can image even failing drives full of bad blocks.

Of course, dd_rescue has its problems as well. While it does skip bad blocks, it takes it some time to do so, and since bad blocks are often grouped together, it can take a very long time to dig through a group of bad blocks. If this group is at the front of your data, you are wasting the valuable last breaths of your hard drive looking at bad blocks, instead of recovering good ones.

The dd_rhelp script addresses this by acting as a front-end for dd_rescue. When dd_rescue finds a bad block, dd_rhelp has it jump ahead a bit, and then read blocks in reverse until it finds the tail end of the bad block group. It then continues forward and grabs all of the good blocks first. Then, after it finishes grabbing your good data, it goes back to the bad data and attempts to recover what it can from there.

dd_rhelp has an added advantage in that it can also resume at any point in the recovery by keeping track of where it is in dd_rescue's logs. So if you have to Ctrl-C from the process, you can just re-run it and dd_rhelp will pick up where it left off. Also, it has a nice little ASCII progress meter, which is useful for keeping track of where you are.

Of course, it makes sense to not run this rescue while you are booted into the bad drive, so either move the drive to another system, or use some sort of rescue disk like Knoppix.

So, to use this, download dd_rescue version 1.03 or later (1.03 is also packaged in Debian unstable at the moment, but testing seems to have 1.02) from its site, and tmeat.net/projects/dd_rescue/">http://freshmeat.net/projects/dd_rescue/ (don't confuse it with Garloff's dd_rescue).

ddrescue will:

1) Read the non-damaged part of the disk, skipping the damaged areas.
or
1) Read a list of bad blocks (damaged areas) from a file.

2) Try to read the damaged areas, splitting them into smaller pieces and reading the non-damaged pieces, until the hardware block size is reached.

3) Try to read the damaged hardware blocks until the specified number of retries is reached, or until interrupted by the user.

4) Optionally write a list of bad blocks to a file.

Ddrescue is now part of the GNU project. Its new homepage is http://www.gnu.org/software/ddrescue/ddrescue.html.

[Mar 11, 2010] g4u.

this is bit-by-bit copying so NTFS is supported. It works best with identical size and geometry. People complain that the image resulting from g4u is very big. This is normal as g4u clones the whole disks with all blocks, not attributing if they contain any valid data or if they are empty/unused. To find empty/unused blocks (and not clone them), g4u would need intimate understanding of the contained filesystem, which is different again for each filesystem - Windows FAT, Linux Ext2/3/ReiserFS/..., BSD FFS, Solaris UFS, etc.

Creating a dd-dcfldd Image Using Automated Image & Restore (AIR) HowtoForge - Linux Howtos and Tutorials

Automated Image & Restore (AIR) is an open source application that provides a GUI front end to the dd/dcfldd (Dataset Definition (dd)) command. AIR is designed to easily create forensic disk/partition images. It supports MD5/SHAx hashes, SCSI tape drives, imaging over a TCP/IP network, splitting images, and detailed session logging. To date, the AIR utility has only been developed for use on Linux distributions. In its simplest form, AIR provides a convenient interface to execute the dd set of commands. It eliminates the risk of "fat fingering" an error in the shell terminal and ultimately makes using the dd command more user-friendly for those who are not as experienced. Please note that using the AIR front end still requires some basic knowledge of how the dd (or dcfldd) commands work.

The dd command has been around for quite a while. It is well known throughout the Unix/Linux community, well documented, and as I can only imagine extensively used. A dd image is a bit by bit image of a source device or file. The uses for dd range from creating and maintaining system backups and restore images to the forensic application of imaging evidence that will be returned to the lab and examined.

This tutorial is not designed to teach the use of the dd command; this is well documented and a simple internet search will yield a plethora of results. Instead, the intent of this mini "how-to" is to introduce users to the AIR front end application, increase overall awareness of the utility, and provide a brief example of creating a dd image using this tool.

DISCLAIMER: I do not claim to be expert at using dd or Automated Image & Restore.

dd_rescue Advantages

Imagine, one of your partitions is crashed, and as there are some hard errors, you don't want to write to this hard disk any more. Just getting all the data off it and retiring it seems to be suitable. However, you can't access the files, as the file system is damaged.

Now, you want to copy the whole partition into a file. You burn it on CD-Rom, just to never lose it again. You can setup a loop device, and repair (fsck) it and hopefully are able to mount it.

Copying this partition with normal Un*x tools like cat or dd will fail, as those tools abort on error. dd_rescue instead will try to read and if it fails, it will go on with the next sectors. The output file naturally will have holes in it, of course. You can write a log file, to see, where all these errors are located.

The data rate drops very low, when errors are encountered. If you interrupt the process of copying, you don't lose anything. You can just continue at any position later. The output file will just be filled in further and not truncated as with other Un*x tools.

If you have one spot of bad sectors within the partition, it might be a good idea, to approach this spot from both sides. Reverse direction copy is your friend.

The two block sizes are a performance optimization. Large block sizes result in superior performance, but in case of errors, you want to try to salvage every single sector. So hardbs is best be set to the hardware sector size (most often 512 bytes) and softbs to a large value, such as the default 16k.

Install dd_rescue in Debian

Install ddrescue using the following command

#apt-get install ddrescue

Install ddrescue in Ubuntu

sudo apt-get install ddrescue

This will complete the installation

ddrescue Syntax

dd_rescue [options] infile outfile

Now we will see how to use ddrescue under damaged disk

If you have a damaged hard disk /dev/sda1 and you have an empty space hard disk /dev/sda2 Now if you want to copy data from /dev/sda1 to /dev/sda2 use the following commnd

# dd_rescue /dev/sda1 /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo dd_rescue /dev/sda1 /dev/sda2/backup.img

This copies an image of /dev/sda1 to sda2

Now you need to check the backup image consistency this will check for is there any problems with this image.

#fsck -y /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo fsck -y /dev/sda2/backup.img

After finishing this checking you need to mount your disk image in to your other hard disk

#mount /dev/sda2/backup.img /mnt/recoverydata

If you are using ubuntu linux use the followingprebackup.img /mnt/recoverydata

This will mount all the data from the backup.img under /mnt/recoverydata now you can try to access the data it should work without any problem.

Restore image

If you want to restore this image use the following command

#dd_rescue /dev/sda2/backup.img /dev/sda1

If you are using ubuntu linux use the following command

sudo dd_rescue /dev/sda2/backup.img /dev/sda1

Copy Disk Image to remote machine using SSH

If you want to copy your disk image to remote machine over ssh you need to use the following command

#dd_rescue /dev/sda1 - | ssh username@machineip 'cat /datarecovery/backup.img'

If you are using ubuntu linux use the following command

sudo dd_rescue /dev/sda1 - | ssh username@machineip 'cat /datarecovery/backup.img'

This will be prompetd for password of the username you have menctioned in the above command after entering the password dd_rescue strats copying obviously it will take some time to copy over the network.

Possible Error

If you see the following error at the time of copying you can ignore this error

dd_rescue: (warning): output file is not seekable!
dd_rescue: (warning): Illegal seek

If you want to take this image in compressed format you can use the following command format

#tar zcvf - /dev/sda1 | ssh username@machineip 'cat@@>/tmp /datarecovery/backup.tar.gz'

If you are using ubuntu linux use the following command

sudo tar zcvf - /dev/sda1 | ssh username@machineip 'cat@@>/tmp /datarecovery/backup.tar.gz'

If you want to know more available options check dd_rescue man page

recover-data-from-a-dead-hard-drive-using-ddrescue

Like dd, dd_rescue does copy data from one file or block device to another.dd_rescue is a tool to help you to save data from crashed partition. It tries to read and if it fails, it will go on with the next sectors where tools like dd will fail. If the copying process is interrupted by the user it is possible to continue at any position later. It can copy backwards.

[Feb 15, 2007] Recover Data from a Damaged hard disk using dd_rhelp

Debian Admin

December 19, 2006 (debianadmin.com) by Admin @ 4:59 pm. Filed under Backup

dd_rhelp is a bash script that handles a very usefull program written in C which is called dd_rescue, it roughly act as the dd linux command with the characteristic to NOT stop when it falls on read/write errors.

This makes dd_rescue the best tool for recovering hard drive having bad sectors. In short, it'll use dd_rescue on your entire disc, but will try to gather the maximum valid data before trying for ages on badsectors. So if you leave dd_rhelp work for infinite time, it'll have the same effect as a simple dd_rescue. But because you might not have this infinite time , dd_rhelp will jump over bad sectors and rescue valid data. In the long run, it'll parse all your device with dd_rescue.

You can Ctrl-C it whenever you want, and rerun-it at will, it'll resume it's job as it depends on the log files dd_rescue creates.In addition, progress will be shown in a ASCII picture of your device beeing rescued.

Why use dd_rhelp and not dd_rescue ?

dd_rhelp uses dd_rescue to compute a recovery path through the device that will focus on valid data recovering. This recovery path will go through all the device, exactly as dd_rescue could do it on its own without any path. This means that dd_rhelp will save you time ONLY IF YOU INTEND TO CANCEL ITS JOB BEFORE THE END of a full recovery.

Because a considerable amount of time is taken to try to rescue badsectors. This amount of time can be mesured in days,
month, years, depending on your device capacity and its defectiveness. You might not want to spend this time knowing that 99 percent of this time will be taken to look at badsector and won't lead to any more data recovering.

dd_rhelp shifts this useless waiting time to the end. Using dd_rescue strait throughout your device make your waiting time dependent on the badsector distribution.

Install dd_rhelp

First you need to download latest version from here

Preparing your system

#apt-get install make gcc g++ ddrescue

If you are using ubuntu linux use the following command

sudo apt-get install make gcc g++ ddrescue

Download dd_rhelp using the follwoing command

wget http://www.kalysto.org/pkg/dd_rhelp-0.0.6.tar.gz

Now you have dd_rhelp-0.0.6.tar.gz file

Extract this file using the following command

tar xzvf dd_rhelp-0.0.6.tar.gz

cd dd_rhelp-0.0.6

Run the following commands

# ./configure

# make

# make install

If you are using ubuntu linux use the following command

sudo ./configure

sudo make

sudo make install

This will complete the installation and this will install program under /usr/local/bin/dd_rhelp

dd_rhelp Syntax

dd_rhelp {filename|device} {output-file} [{info}]
or dd_rhelp --help
or dd_rhelp --version

Now we will see how to use ddrescue under damaged disk

If you have a damaged hard disk /dev/sda1 and you have an empty space hard disk /dev/sda2 Now if you want to copy data from /dev/sda1 to /dev/sda2 use the following command

#dd_rhelp /dev/sda1 /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo dd_rhelp /dev/sda1 /dev/sda2/backup.img

Output looks like below

/usr/bin/tail: Warning: "+number" syntax is deprecated, please use "-n +number"
=== launched via 'dd_rhelp' at 0k, 0 >>> ===
dd_rescue: (info): ipos: 104384.0k, opos: 104384.0k, xferd: 104384.0k
errs: 0, errxfer: 0.0k, succxfer: 104384.0k
+curr.rate: 10178kB/s, avg.rate: 8532kB/s, avg.load: 31.3%
dd_rescue: (info): /dev/sda2 (104391.0k): EOF
Summary for /dev/sda2 -> backup.img:
dd_rescue: (info): ipos: 104391.0k, opos: 104391.0k, xferd: 104391.0k
errs: 0, errxfer: 0.0k, succxfer: 104391.0k
+curr.rate: 355kB/s, avg.rate: 8519kB/s, avg.load: 31.3%
/usr/bin/tail: Warning: "+number" syntax is deprecated, please use "-n +number"

This copies an image of /dev/sda1 to sda2

Now you need to check the backup image consistency this will check for is there any problems with this image.

#fsck -y /dev/sda2/backup.img

If you are using ubuntu linux use the following command

sudo fsck -y /dev/sda2/backup.img

After finishing this checking you need to mount your disk image in to your other hard disk

#mount /dev/sda2/backup.img /mnt/recoverydata

If you are using ubuntu linux use the following command

sudo mount /dev/sda2/backup.img /mnt/recoverydata

This will mount all the data from the backup.img under /mnt/recoverydata now you can try to access the data it should
work without any problem.

Restore image

If you want to restore this image use the following command

#dd_rhelp /dev/sda2/backup.img /dev/sda1

If you are using ubuntu linux use the following command

sudo dd_rhelp /dev/sda2/backup.img /dev/sda1

Copy Disk Image to remote machine using SSH

If you want to copy your disk image to remote machine over ssh you need to use the following command

#dd_rhelp /dev/sda1 - | ssh username@machineip 'cat /datarecovery/backup.img'

If you are using ubuntu linux use the following command

sudo dd_rhelp /dev/sda1 - | ssh username@machineip 'cat /datarecovery/backup.img'

This will be prompetd for password of the username you have menctioned in the above command after entering the password

dd_rhelp strats copying obviously it will take some time to copy over the network.

Recommended Links

• Ddrescue - GNU Project - Free Software Foundation (FSF)

• Ddrescue - Forensics Wiki

Softpanorama Recommended

Top articles

Sites

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019