Working with dd Images

The most important innovation of recent years was introduction of  Loopback filesystem which permit mounting DD-images as regular partitions.  That permits per file restoration which was always a strong point of file based backups. Also with snapshot capabilities of modern filesystem it is possible to take DD image of a "live" partition. Previously you need to boot from another, typically CD or DVD image to perform such a backup.

Another important new class of tools are so called file carvers. One example of such a tool is Scalpel:

Scalpel is a fast file carver that reads a database of header and footer definitions and extracts matching files or data fragments from a set of image files or raw device files. Scalpel is filesystem-independent and will carve files from FATx, NTFS, ext2/3, HFS+, or raw partitions. It is useful for both digital forensics investigation and file recovery.

Notes on Platforms

Linux

The preferred platform for using Scalpel is Linux.

Windows

Scalpel will also compile under Windows (32 or 64-bit) using mingw. If you'd like to try Scalpel on Windows without the bother of compiling it yourself, an executable and appropriate libraries are included in the distribution--just untar and go. Note that under Windows, the pthreads DLL must be present in the same directory as the Scalpel executable. Carving physical and logical devices directly under Windows (e.g., using \\.\physicaldrive0 as a target) is not supported in the current release.

Mac OS X

As of v1.53, Scalpel is supported on Mac OS X.

All platforms

As of v1.54, Scalpel supports carving files larger than 4GB on all platforms.

As of v1.60, Scalpel supports preview carving and other new carving modes. See the distribution for details.

As for v2.0, Scalpel supports regular expressions for headers and footers, minimum carve sizes, multithreading and asynchronous I/O, and beta-level support for GPU-accelerated file carving.

NEWS CONTENTS

• 20120105 : Scalpel: A Frugal, High Performance File Carver ( digitalforensicssolutions )
• 20110726 : Creating and using disk images mini-howto Marcs Realm ( Creating and using disk images mini-howto Marc's Realm, Jul 26, 2011 )
• 20110726 : Re how to write a floppy using dd ( Re how to write a floppy using dd, )
• 20110726 : Moving your data to a backup device ( Moving your data to a backup device, )
• 20110726 : Making Diskette Images without Diskettes ( Making Diskette Images without Diskettes, )

Old News ;-)

[Jan 05, 2012] Scalpel: A Frugal, High Performance File Carver

digitalforensicssolutions

Scalpel is a fast file carver that reads a database of header and footer definitions and extracts matching files or data fragments from a set of image files or raw device files. Scalpel is filesystem-independent and will carve files from FATx, NTFS, ext2/3, HFS+, or raw partitions. It is useful for both digital forensics investigation and file recovery.

Notes on Platforms

Linux

The preferred platform for using Scalpel is Linux.

Windows

Scalpel will also compile under Windows (32 or 64-bit) using mingw. If you'd like to try Scalpel on Windows without the bother of compiling it yourself, an executable and appropriate libraries are included in the distribution--just untar and go. Note that under Windows, the pthreads DLL must be present in the same directory as the Scalpel executable. Carving physical and logical devices directly under Windows (e.g., using \\.\physicaldrive0 as a target) is not supported in the current release.

Mac OS X

As of v1.53, Scalpel is supported on Mac OS X.

All platforms

As of v1.54, Scalpel supports carving files larger than 4GB on all platforms.

As of v1.60, Scalpel supports preview carving and other new carving modes. See the distribution for details.

As for v2.0, Scalpel supports regular expressions for headers and footers, minimum carve sizes, multithreading and asynchronous I/O, and beta-level support for GPU-accelerated file carving.

[Jul 26, 2011] Creating and using disk images mini-howto Marc's Realm

Table of contents [hide]

1. Introduction
2. Creating a disk image
3. Mounting the disk image
   1. The clean way
   2. The dirty way
4. Final comments
5. Credits

Introduction

I had the need of booting Linux from a compact flash card in an embedded device. And for producing small quantities in-house it was necessary to have disk images from these flash cards that could just be dd'ed onto the the flash cards. Since it took me some time to figure out how to do that I've written this guide in hope others may find it useful.

Our first method to get these disk images was to boot the embedded device via Knoppix, mount the compact flash, copy the necessary files via scp onto the compact flash, chroot onto the flash, run LiLo and reboot. Afterwards we'd dd the complete flash content into a file. Not very entertaining. And especially hard to automate when you have a new release very often.

So we wanted to automate the image creating process as much as possible. Using an USB CF reader/writer we thought this shouldn't be too hard, but it turned out that when we copied the files onto the flash and chroot'ed into it lilo refused to run (can't remember why, sorry).

So we got the idea of producing bootable mini-images, where we would mount the partition using the loopback device, copy the files in, unmount the image and dd that complete image onto the compact flash (complete with MBR, partition table, everything).

Problem is, again lilo is making problems: you can't just update the kernel by copying a new one over the old one. You have to run lilo again. And grub was out since neither of us managed to get it to work (while grub seems to be very good, the configuration is an unnecessarily hairy nightmare Almost the same technique as described in this mini-howto can be use with grub if you dig it. And saves you the FAT16 partition as well.). Alternatives: booting DOS, using LOADLIN or SYSLINUX. Obviously, SYSLINUX is the cleaner solution.

This simple technique described can also be used with any other medium, like USB sticks for example.

Creating a disk image

And this is how you do it:

1. Insert CF into reader/writer. We assume that the CF is now accessible as /dev/sda.
2. Since our raw CF's had lots of garbage on it we zero out the complete CF (helps compressing the image later on We used 64MB CF's when I wrote the first version of this documents. When zero'd, partitioned and formated these compressed down to just 4200 bytes with bzip2... nice ratio :-) An even nicer ratio is that of a 2GB hard disk image we've done: it compresses from 2GB down to just 18613 bytes.).

   dd if=/dev/zero of=/dev/sda

3. Create partitions: we need at least one boot partition (FAT12 or FAT16, but not FAT32) and a root partition (we used Ext3).

   fdisk /dev/sda

4. Format the partitions.

   mkfs.msdos /dev/sda1
   mkfs.ext3 /dev/sda2

5. Install SYSLINUX on boot partition.

   syslinux -s /dev/sda1

6. Install master boot record (found in SYSLINUX source directory).

   dd if=mbr.bin of=/dev/sda

7. Mount the boot partition.

   mount /dev/sda1 /mnt

8. Copy the kernel image onto boot partition.

   cp bzImage /mnt/kernel.bzi

9. Create SYSLINUX configuration file.

   cat >/mnt/syslinux.cfg <<"EOF"
   DEFAULT kernel

   LABEL kernel
   KERNEL kernel.bzi
   APPEND root=/dev/hdc2
   EOF

10. Umount the boot partition.

    umount /mnt

11. Save the final image.

    dd if=/dev/sda of=image.bootable

You can then mount the root directory and copy all your files into it, and even update the kernel by just copying a new bzImage onto the boot partition. No need to run any program like LiLo afterwards.

If you just want to copy the partitioned space then you may want to read on about mounting the disk image and then come back here: you need to calculate the size, which is (<end block number of the last partition> + 1) * 512. Then give dd the additional option count=<size>.

Mounting the disk image

There are two ways to mount the partition.

The clean way

First, we need to determine the offset of the partition. This is quite easy: just type fdisk -ul <device>. The option -ul means list the partitions on the device and assume a unit size of 512 byte. This looks something like this:

tetsuo:~ # fdisk -ul /dev/sda

Disk /dev/sda: 256 MB, 256376832 bytes
8 heads, 62 sectors/track, 1009 cylinders, total 500736 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 62 19839 9889 4 FAT16 <32M
/dev/sda2 19840 231135 105648 83 Linux
/dev/sda3 231136 442431 105648 83 Linux
/dev/sda4 442432 471199 14384 83 Linux

Now all we need to do is a little math to get the offset: we need to multiply the start block by 512. E.g. if we wanted to mount the first partition we'd have an offset of 62 * 512 = 31744. The second partition has an offset of 19840 * 512 = 10158080. Now that we have the offset we can mount the partition:

mount -o loop,offset=10158080 image.bootable /mnt

This would mount the second partition on /mnt. Linux recognizes it as ext3 if it is formatted as ext3 and the kernel supports ext3, so no need for a -t ext3 option to mount.

The dirty way

There is also a hard way to find the formatted partitions if you can't calculate the offsets for some reason:

for ((i=0 ; $i < 10000 ; i=$i + 1)) ; do
mount -o loop,offset=$(($i * 512)) image.bootable /mnt && break
done

If there is a partition within the first 10000 blocks, it gets mounted eventually :-) Just type "mount" to get the offset...

Final comments

After we've unmounted the disk image we can now just dd the disk image to a new compact flash:

dd if=image.bootable of=/dev/sda

Easy as that.

There are several ways to force Linux to re-read the partition table after we've written a disk image with partition table to an empty compact flash. Propably the best way is to run:

partprobe

This program is part of GNU parted. If it's not installed then you might succeed with the following command:

/sbin/sfdisk -R /dev/sda

In the rare case that you have neither, there's still a hack: unload the USB module and load it again:

modprobe -r usb-uhci && modprobe usb-uhci

Credits

This document was written and is ©opyrighted 2003,2006,2010 by Marc Haisenko. Thanks to the SYSLINUX author H. Peter Anvin for finding an unnecessary step in the creation process. This moved to the chapter "Final comments". Manicalic told me about partprobe and sfdisk to re-read the partition table. If you have further comments/additions/corrections please mail them to me. You may copy and distribute this document as long as you include this credit section and my name. You may modify it and add your name to this section as well.

Re how to write a floppy using dd

There are two basic ways of copying the boot images to floppys.
One is by using dd: dd if=1440_boot_floppy of=/dev/fd0 -- of course
use your own intended floppy device.

The second might be a little quicker:

cat 1440_boot_floppy >/dev/fd0

I have used both ways at different times and they work the same.
There is probably quite a tech. difference, but I don't think that there
is a functional difference.

Have Fun!

Moving your data to a backup device

Using the dd command to dump data

The dd command can be used to put data on a disk, or get it off again, depending on the given input and output devices. An example:

gaby:~>dd if=images-without-dir.tar.gz of=/dev/fd0H1440
98+1 records in
98+1 records out

gaby~>dd if=/dev/fd0H1440 of=/var/tmp/images.tar.gz
2880+0 records in
2880+0 records out

gaby:~>ls /var/tmp/images*
/var/tmp/images.tar.gz

Note that the dumping is done on an unmounted device. Floppies created using this method will not be mountable in the file system, but it is of course the way to go for creating boot or rescue disks. For more information on the possibilities of dd, read the man pages.

This tool is part of the GNU fileutils package.

Dumping disks

The dd command can also be used to make a raw dump of an entire hard disk.

Making Diskette Images without Diskettes

So far we have been preparing boot diskettes by writing to real diskettes. This sounds like the most logical way to do it, but there can be reasons why we want to prepare an image file of a diskette without using real diskettes. Several reasons could be:

• Create a diskette image for a diskette drive you do not have.
• Automate boot image creation (several diskette images for a distribution).
• Create a diskette image for bootable CD-ROM
• Create a diskette image for a PC emulator

Basically we could create a diskette image as follows:

• Create an all zero image file using dd.
• Create a file system onto the image file.
• Mount the image file using the loop option and copy all files to it.
• Install the boot loader onto the image file.

The last part is the trickiest especially for LILO. It's fairly trivial for SYSLINUX and using the device command it can be done with GRUB. There is also another trick for SYSLINUX and GRUB (it does not work with LILO):

• Start with an image file with just the boot loader installed and an empty file system and copy that image file each time you create another image. This image file may be extracted from a real diskette just once.
• Mount the image file using the loop option and copy all files to it.

On Timo's Rescue CD Page there is a good explanation of how to create 2.88MB diskette images for a bootable CD-ROM, using all boot loaders. I could not explain it better. Of course these recipes apply also to other types of disk images.

Recommended Links

Softpanorama Recommended

Top articles

Sites

Comparison of disk cloning software - Wikipedia, the free encyclopedia

Marc's realm - Creating and using disk images mini-HOWTO

Thomas Rude - DD and Computer Forensics - He's Worth a Deuce!

**** Rawrite and related programs very good page by Jeremy Davis

Marc's realm - Creating and using disk images mini-HOWTO

Rawrite Wikilearn TWiki

Thomas Rude - DD and Computer Forensics

Using dd (instead of cdrecord and dvdrecord) to write bootable ISO images to DVD-RAM disks - CCP14 Linux Internet Server - CCP14 Web-Config Administration Information - CCP14 Homepage - Single Crystal and Powder Diffraction

Rawrite

raw write program looks like version 1.3 is the latest. It's a single exe file

rawrite -- Suse archive contain rawrite.exe and rawrite3.com: the latter has parameters (-d -f) There is also rawrite2 program written in Pascal looks like semidebugged (not working under Windows 2000) analog of wimage. Because of problems with win2000 does not have any advantages over wimage.

DCF: Disk Copy Fast This DOS shareware tool can read/write/format image files compatible with WinImage under MS-Dos. Works under Win2K. ftp://ftp.simtel.net/pub/simtelnet/msdos/diskutil/dcf5_3.zip
http://ourworld.compuserve.com/homepages/dcf_hdcp/DCF53.ZIP

** ? WinImage WinImage looks like a wimage based utility. Too complex to be useful.

Softlookup.com - Floppy Image Creator-Display Information

Floppy Image Creator This utility can work with any of the standard floppy disk formats: 720kb and 1.44Mb 3.5" disks, as well as 360kb and 1.2Mb 5.25" disks. You can also include a description of the disk with the image file.

Muckshifter's Forum - Image Maker 1.1 free

The FREE ImageMaker 1.1 does not support image compression and encryption. These features will be added in a future shareware version of ImageMaker. The image of an entire hard drive may be restored only to a hard drive, not a partition, and vice versa. After restoring a disk partition backup, a reboot is required to see the restored disk contents. After restoring entire hard disk backups, or in any other cases, no reboot is requred. The download is very small, at just 476KB.

DCF

DCF: Disk Copy Fast This DOS shareware tool can read/write/format image files compatible with WinImage under MS-Dos. Works under Win2K. ftp://ftp.simtel.net/pub/simtelnet/msdos/diskutil/dcf5_3.zip
http://ourworld.compuserve.com/homepages/dcf_hdcp/DCF53.ZIP

Wimage

Good old wimage (part of FDFORMAT, a shareware package for DOS written by Christoph H. Hochsttter) can still be useful.

• Index of -pascal-UNITS-FDFORM
• Evseev's download page

Random Findings

Muckshifter's Forum - Image Maker 1.1 free

The FREE ImageMaker 1.1 does not support image compression and encryption. These features will be added in a future shareware version of ImageMaker. The image of an entire hard drive may be restored only to a hard drive, not a partition, and vice versa. After restoring a disk partition backup, a reboot is required to see the restored disk contents. After restoring entire hard disk backups, or in any other cases, no reboot is requred. The download is very small, at just 476KB.

Rundegren.com - Floppy Image OK program but will cost you $15. There are some free older versions.

Create image files of floppy disks and back (for backup, shipping or transfer). Save the image file compressed, uncompressed or as a self-extracting exe. Add descriptions to or convert your old image files. Supports DMF and other non-standard formats. Logically recreate bad sectors when writing floppy disks, allowing for an exact duplicate of your source disk to be created. The self-extracting exe can be fully customized with a picture, license agreement dialog and a text with instructions. Full support for drag-n-drop.

NOTE: Formatting of non-standard formats and recreating bad sectors logically are only available on Windows NT4/2000/XP.

Softlookup.com - Floppy Image Creator-Display Information

Floppy Image Creator This utility can work with any of the standard floppy disk formats: 720kb and 1.44Mb 3.5" disks, as well as 360kb and 1.2Mb 5.25" disks. You can also include a description of the disk with the image file.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019