Cloning harddrives and partitions using dd

A disk image is a file which contains the full contents of the specified disk or partition in a single file; the contents are generally read as raw sectors so all data including partition information (FAT), boot sector, along with actual directory entries and files are copied to the image.

The first widespread use of raw images was images of floppy drives. They were device specific: a 1.44MB image should be written to a 1.44MB floppy diskette and 1.2MB image should be written to a 1.2MB floppy diskette. However, soon more advanced programs started to support cross diskette writing (i.e. 1.2MB disk image to a 1.4MB disk drive).

The same is true with disk images. Typically raw image made using dd can be written to hard drives of the same or larger size [later adjusting  by resizing partition(s) and take advantage of additional size].  In modern Linux you can mount dd-image using loopback interface and not only images of ext3 and other linux filesystem but partitions with NTFS as well, for example:

mount -t ntfs -o loop,ro,umask=0222 ./nonbootable.dd /media

A disk image is different from an archive [zip/arj/etc.] as it contains a snapshot (or image) of the contents and does not differentiate between OS data (FAT/etc.) and user data (particular files/directories/etc.); which are not generally accessible until written back to a disk; and use different means to access the data bypassing the OS's filesystem layer; also a diskimage will generally produce identical disk copies

Unix DD has some advantages over Ghost and similar tools. It blindly copies all bytes from source to target and due to this can help to recover lost files from almost any filesystem. You just need to create a DD image and then search it for files using grep or similar tool. In most cases you can recover files pretty reliably using this method. I one manage to recover a script that was deleted several hours ago before it was discovered and server continued working all this time. That was the only copy and the user who did it has no backup.  See Recovery of lost files using DD

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

Copying Windows to a new drive, using linux - How-to/Guide

Written 20 Mar 2005 by Ed Anderson. Updated 18 May 2009 (NTFS Updates; device names)

Overview

This guide will show you how to copy an existing installation of Windows (or any other OS) from one drive to another - as long as the destination drive is the same size or larger.

This is a free and relatively easy method that will create a clone of your current hard disk, without having to buy any third party software.

1. Gathering tools
2. Physical Installation
3. Preparing new partition table
4. Copy the MBR
5. Copy the Partition
6. Resizing the Partition
7. FAQ

Gathering Tools

For this project, you will need a Linux Live CD that has dd, fdisk, parted/qtparted, and ntfsresize.
Knoppix is a Live CD that has everything you'll need. Many other Live CDs will work as well, such as an Ubuntu install CD.

Burn the contents of the ISO to CD to boot from later.

Physical Installation

Install both hard drives in your computer. In this guide, I will use the following disk setup as an example:

/dev/sda (Primary Master) - New, empty 80G drive
/dev/sdb (Primary Slave) - Old 10G drive with all data on one NTFS partition (/dev/sdb1)

Your disk setup can vary, but in this guide, I will refer to the new drive as /dev/sda, and the old as /dev/sdb. Use device names according to your own setup. For example, yours might be /dev/sda and /dev/sdb.

If you're not familiar with this naming scheme for disks/partitions, read the Linux Partition HOWTO.
To sum it up, SCSI disks are referred to by /dev/sdX where X is corresponds to the disk order (a for the first disk, d for the last). Partitions are referred to as /dev/sdXY where Y is the partiton number. (eg. /dev/sdc2 is the 3rd disk, 2nd partition) IDE disks are the same, but hdX instead of sdX.

Preparing new partition table

Before you start, you may want to run scandisk on your drive to verify that there are no errors on the disk.

Boot from the Live CD. Press enter at the boot: prompt.

Open a new terminal as root. In knoppix 3.7, click on the Penguin icon, and click Root Shell.

View the partition table for the old disk:

# fdisk -l -u /dev/sdb

Disk /dev/sdb: 10.2 GB, 10240000000 bytes
255 heads, 63 sectors/track, 1244 cylinders, total 20000000 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          63    19968794     9984366    7  HPFS/NTFS

Note that I used the -u option on fdisk. This displays the start and end units in sectors, rather than cylinders, since the cylinder size varies from disk to disk.

Record the Start and End positions, and the Id for the existing partition.

Now run fdisk on the new disk.

# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or
OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 158730.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by
w(rite)

Command (m for help):

When you run fdisk on an empty drive, it may tell you that there is no partition table, or that the disk is large. You may safely ignore these warnings.

Print the current partition table to verify that there are no partitions on the disk.

Command (m for help): p

Disk /dev/sda: 81.9 GB, 81920000000 bytes
16 heads, 63 sectors/track, 158730 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

   Device Boot      Start         End      Blocks   Id  System

If there are partitions listed, and you were expecting none, make sure you ran fdisk on the correct device. You should delete the existing partitions on the new disk, only if you know that the data is not needed.

Create an identical partition, using the Start and End positions from the other disk. Be sure to change the units to Sectors.

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p

Partition number (1-4): 1
First sector (63-159999999, default 63): 63
Last sector or +size or +sizeM or +sizeK (63-159999999, default 159999999): 19968794

Command (m for help): p

Disk /dev/sda: 81.9 GB, 81920000000 bytes
16 heads, 63 sectors/track, 158730 cylinders, total 160000000 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63    19968794     9984366   83  Linux

Before we're done, we must set the Boot flag and System Id. Use the same Id that was listed on your old partition table.

Command (m for help): a
Partition number (1-4): 1

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 7
Changed system type of partition 1 to 7 (HPFS/NTFS)

Command (m for help): p
Disk /dev/sda: 81.9 GB, 81920000000 bytes
16 heads, 63 sectors/track, 158730 cylinders, total 160000000 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63    19968794     9984366    7  HPFS/NTFS

The partition should now look identical to when you ran fdisk -l -u /dev/sdb

Now we must write the changes to disk.

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Copy the MBR

For the new disk to boot, we must copy the boot code from the Master Boot Record (MBR) to the new disk.

The MBR is on the first sector of the disk, and is split into three parts:

• Boot Code (446 bytes)
• Partition Table (64 bytes)
• Boot Code Signature = 55aa (2 bytes)

We only want to copy the boot code - the first 446 bytes. We do this with dd:

# dd if=/dev/sdb of=/dev/sda bs=446 count=1
1+0 records in
1+0 records out
446 bytes transferred in 0.026312 seconds (16950 bytes/sec)

Copying the Partition

Optional: We should try to enable DMA on both disks, to increase the transfer speed.

# hdparm -d 1 /dev/sda
/dev/sda:
 setting using_dma to 1 (on)
 using_dma    =  1 (on)

# hdparm -d 1 /dev/sdb
/dev/sdb:
 setting using_dma to 1 (on)
 using_dma    =  1 (on)

The next task is to copy the filesystem from one disk to the other. We'll do this with dd as well. This time, instead of working with the disk device (hda, hdb), we'll be using the partition device (hda1, hdb1).

# dd if=/dev/sdb1 of=/dev/sda1 bs=4096
2496092+0 records in
2496092+0 records out
10223990784 bytes transferred in 355.312 seconds (28774684 bytes/sec)

This may take a long time, especially if the partition size is large. You can verify that it's doing "something" by running top in another terminal.

Resizing the Partition

Next, we resize the partition to fill the disk. You should only do this section if you are copying to a larger disk, and don't want to leave unpartitioned space.

By far, the easist way is with qtparted. It is a graphical front-end to parted and ntfsresize. If you are using a Live CD without a GUI, you'll have to use parted or ntfsresize.

Method 1: qtparted

This method should work with most filesystems, including NTFS and FAT32.

1. Start qtparted from the command line.
2. Select the new disk (/dev/sda)
3. Right click on the partition to rezise, and click Resize.
4. Change "Free Space After" to 0, and press OK. The partition should now span the disk.
5. Commit the changes to disk using the the Commit option in the File menu.

After the commit operation finishes with your disk, you can reboot onto your new hard drive, and everything should work. Since you resized your partition, be sure to run scandisk to remove any errors in the newly created filesystem.

Method 2: ntfsresize

(no examples yet)

Method 3: parted

If you don't have a GUI, you can use parted to resize almost any non-ntfs partition. In this example, I'll resize a FAT32 partition to fill the drive.

Run parted. You may safely ignore errors about the partition alignment.

# parted /dev/sda
GNU Parted 1.6.9
Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

Using /dev/sda
Warning: Unable to align partition properly.  This probably means that another
partitioning tool generated an incorrect partition table, because it didn't hav
e
the correct BIOS geometry.  It is safe to ignore,but ignoring may cause
(fixable) problems with some boot loaders.
Ignore/Cancel? i

Display the current partition table (p), and resize the partition to the size of the drive.

(parted) p
Disk geometry for /dev/sda: 0.000-78125.000 megabytes
Disk label type: msdos
Minor    Start       End     Type      Filesystem  Flags
1          0.031   9750.234  primary   fat32       boot, lba
(parted) resize
Partition number? 1
Start?  [0.0308]? 0.0308
End?  [9750.2339]? 78125.000
(parted)

This may take some time, depending on the size of the partition. When it's done, you can reboot into windows. Since you modified the partition, be sure to run scandisk to remove any errors in the newly created filesystem.

FAQ

• Why can't I just copy the files, instead of the whole partition?
  I haven't experimented doing this on Linux. You can't if you have NTFS partitions, since linux can't write files on an NTFS partition.

  On FAT32, I would be afraid of losing windows file attributes, missing files, and messing up file order. The advantage would be that no unused space on the filesystem would need to be transferred. Someone please let me know if you've tried, and I'll update this page.

  If I were to try this method on a FAT32 disk, I would create a large partition on /dev/sda, and run

  # mkfs.vfat -F 32 /dev/sda1
  mkfs.vfat 2.10 (22 Sep 2003)
  # mkdir old new
  # mount /dev/sdb1 /mnt/old
  # mount /dev/sda1 /mnt/new
  # cd old
  # find | cpio -pdV ../new

• Can't I create the MBR with the DOS fdisk /mbr, instead of dd?
  Yes, but that would require making a boot disk, or something else that requires more effort than dd. :) Since we're in the Linux environment, it's easier to use dd.

• I have a suggestion or need some help. Can I contact you?
  Yes. You can find me on IRC on irc.freenode.net #Nilbus, or contact me at nilbus AT nilbus DOT com

---

Comments

geoff
05 Jun 2005, 14:35

Hi there,

I just wanted to let you know that I found your article extremely helpful. I just successfully moved my old drive C: to a new disk using knoppix, dd and qtparted, just as you outlined.

There was one small glitch, however. When I went to boot with the new drive, Windows 2000 was complaining about not having a paging file. Normally, I understand, Windows would attempt to create a new paging file if it can't locate the configured file. However, the issue here is that
the Windows registry didn't recognize the new drive as drive C:. This meant that many settings in the registry that pointed to drive C: (besides just the paging file) were unhappy.

To rectify the problem, I used Microsoft's knowledge base article KB223188
- http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q223188.

It might be helpful to future users to add a link.

Thanks,
-g

Wakko Warner
24 Jul 2005, 20:26

I have had fairly good luck doing this where I work. I did notice a problem with office2000 since it likes to write the 8.3 path names in the registry and if another directory gets there first, it does cause problems which I noticed recently.

I wrote most of the system in perl. I have a 2 stage boot system that loads up my perl based menu system. Stage 1 actually fits on floppy. It searches for the stage 2 which is like a mini-distro that loads into tmpfs via a tar file. Since I deal mostly with windows 2000 and xp, there are problems with hard disk geometry and the fact that the fat32 partition MUST contain the bios boot disk number. It took me several tries to realize this. Also if the heads/sectors in the fat32 boot record don't match the partition table, you will have problems as well. I have a program that installs the boot loader into fat32 for 2000/xp (same thing actually).

I came across your site because I was looking for a way to resize a fat32 partition under linux. (I have an iPod which I was forced to convert from mac hfs+ to vfat due to linux's bad support for hfs+)

Juho Leppamaki
22 Sep 2005, 03:36

Hello!

I would like to suggest adding a dma enable option to your guide. I'm using knoppix 4.0 live-dvd and the copying was really slow. I enabled dma on my old PATA-drive with another root shell, and the copying became much faster. I just typed 'hdparm -d 1 /dev/hda'.

Thanks for the great yet simple guide :)

Joshua Michael Smith
12 Nov 2005, 12:42

Just wanted to say thanks for the guide about copying Windows you have. It's really the best and clearest of its kind that I could find on the web. I was able to rescue WinXP on FAT32 for a friend, with the only problem being that their source drive was dying...which was the reason for the move in the first place.

Again, thanks - the clear writing made it very easy.

Brian Dolan-Goecke
02 Feb 2008, 13:18

Great guide.

I ran across a way to get status on the dd while it is running. (It was in the manpage for dd) If you send it a USR1 signal it will output status on standard out. So you could use these commands to do the copy and output copy stats while it is working. Be sure to use the correct partions.

dd if=/dev/hdb1 of=/dev/hda1 bs=4096& pid=$!
while true ; do kill -USR1 $pid; date; sleep 10; done

That will give you stats every 10 second from the dd command. You will have to stop the while loop by hand (with a ctrl-c in the terminal) after the dd is done. The while loop will continue, but get an error.

Have fun, thanks again.
BTW. It looks like it will take 3 hours to copy a 60GB disk connected up via USB!

magic_ninja
21 Aug 2008, 23:24

well I used this guide to copy a 12 gig partition from my old hard drive onto a new 80 gig for windows, i use linux mostly but I use windows for games, this guide was really easy as I have not used dd before, keep up the good work

Gennady
14 Oct 2008, 23:20

I've spent several hours trying to copy a bootable NTFS drive with Windows XP (6 GB) to a larger one (20 GB) using the commercial Drive Image program. I was not able to do it due to various problems. After having found your tutorial, I did the job in 20 min.! Thanks a lot.

steve
30 Dec 2008, 16:38

Very cool directions...

The only issue I had is that after resizing using qtparted and rebooting, I got a no system disk or disk error message. I had to use Partition magic to set the NTFS partition to \"active\" and then it booted right up.

These directions are a life-saver!!!!

cheers

Frank Parker
09 Jan 2009, 00:11

I successfully cloned windows ME from a 20 gig drive to a 500 gig drive. The disk was formatted in fat 32. The only issue is that I used knoppix 5.1 which demanded more memory then the computer had. I should have used an older version of knoppix.

Thanks for the help!

TexasEric
23 Jan 2009, 14:16

Great directions! I would suggest users try and use dd_rescue as opposed to the older dd. dd_rescue has the ability to grab larger and larger slices until it finds a disk error. When it does find an error, it shrinks the slices down smaller and smaller so that only the 'damaged' portions are skipped. It is a great little program.

Randall Wayth
13 Mar 2009, 21:28

Just wanted to say thanks for the excellent page on copying a windows installation to a new disk. I have just done this to a new, larger hard disk in my laptop where I went from a 10G NTFS windows to a 15G NTFS.

It might be worth updating the page regarding the caveats about NTFS. Writing to NTFS partitions has been supported for a few Ubuntu releases now. Furthermore, copying from a smaller->larger partition is super easy using ntfsresize.

Recipe is:
create new partition
copy, via dd, contents of old to new
ntfsresize new

Windows did a disk check when I booted into it after the copy/resize, but otherwise everything went fine.

Cheers

Boes
25 Jun 2009, 13:59

Thanks.

I burned a SystemRescueCd cd from www.sysresccd.org and followed your directions with succes.

The disk was partitioned with a primary partition (sda1, windows disk C) and an extended partition (sda2). In the extended partition another NTFS partition existed (sda5, windows disk D). On the new disk I created the same partitions, where the extended partition can get a size greater than or equal the extended partition on the old disk. I only needed to dd copy the MBR, sda1 and sda5 to the new harddisk.

Restarted the computer without the old harddisk and now everything is up and running!

Perlakowski
26 Jun 2009, 18:56

absolutely gorgeous. just wonderful. :),

Stephen Winnall
09 Jul 2009, 15:38

Thanks for this excellent article. It worked a treat when I copied a Vista system from a 500 GB to a 1000 GB disk.

There was one drawback though. The copy insisted that it was the E: disk when I booted from it and apparently tried to referenced stuff on the C: disk, which was not good at all when I removed the C. disk completely from the computer.

The following article helped me to resolve that problem by making the copy boot as the C: disk:

http://www.multibooters.co.uk/articles/drive_letters.html

Norman Ramsey
26 Jul 2009, 22:39

Thanks for the helpful tutorial. I used ntfsclone instead of dd, and after running qtparted I had to run ntfsresize to get not only the partition but also the filesystem up to full size. But everything's great and I'm now having windows check the new disk for bad sectors (full hardware scan).

Thanks a lot for the helpful tutorial.

Lazyfirecloud
25 Aug 2009, 18:57

Thank you so much! This tutorial has been very helpful. The MBR always ends up screwing me =)

I personally used gparted to copy and resize each partition's content over; it uses ntfsclone and ntfsresize, as well as checks the filesystems for errors before/after each operation. It's quite the tool =)

Lazyfirecloud
25 Aug 2009, 18:57

Thank you so much! This tutorial has been very helpful. The MBR always ends up screwing me =)

I personally used gparted to copy and resize each partition's content over; it uses ntfsclone and ntfsresize, as well as checks the filesystems for errors before/after each operation. It's quite the tool =)

Steve Brodie
14 Oct 2009, 16:54

Very interesting guide, and much needed since clear information on how to do this with Free and Open Source Software (FOSS) seems to be hard to find.

I am curious about the 'dd if=/dev/sdb of=/dev/sda bs=446 count=1' instruction: is the 'bs=446' specific to the windows OS, or would the same '446' number be applicable to linux systems? I ask because typically I see the number '512' used in other instructions about how to backup an MBR; maybe this number is applicable to linux?

Can anyone confirm?

Thanks,
SB

Ed Anderson
14 Oct 2009, 17:10

466 excludes the last 46 bytes, which is where the partition table is stored. Since tha partition table would differ on disks of different geometry and size, it's probably better to recreate the partition table using the right tools rather than copying the source disk's partition table.

Steve Brodie
15 Oct 2009, 10:19

Ed Anderson> 466 excludes the last 46 bytes, which is where the partition table is stored. Since tha partition table would differ on disks of different geometry and size, it's probably better to recreate the partition table using the right tools rather than copying the source disk's partition table.

So if I understand you correctly you are saying that:
1. It is better to use '446' rather than '512';
2. It is always better to do so (in this context anyway);
3. The '466' value is a constant that will always apply, regardless of what OS one is working with.

Is this correct?

Thanks,
SB

Ed Anderson
15 Oct 2009, 11:28

Sorry, I typo'd the number and then used it in my math. :P It's 446 bytes, leaving 64 bytes for the partition table and a 2 byte signature. See http://en.wikipedia.org/wiki/Master_boot_record

Yes, this is OS independent.

Following this guide, you would not want to copy all 512 bytes of the boot sector to the new disk, because it would copy the partition table from the old disk and write over the partition table you had just created on the new disk.

Different disks will usually have different disk geometry (number of cylinders, heads, sectors). Since the partition table stores partition locations using these values, I don't know what the behavior would be if you copied the partition table from one disk to another of a different size.

G. Miller
16 Jan 2010, 09:07

Thank you for the excellent guide. This is an extremely slick and easy way to copy a system. I was able to copy the system without any problems, but had some difficulty with the partition resize. I tried 4 or 5 different Linux LiveCDs, but couldn't get gparted to work (160GB SATA drive, ntfs). It kept reporting a bad sector problem that didn't seem to actually exist (ran chkdsk several times, no problems). I poked around the net and ran across EASEUS which I downloaded (free for home use) and it seemed to do the trick.

I really appreciate the work you put into this.

GM

Flyte
25 Jan 2010, 21:21

Excellent tutorial - followed it and it worked perfectly, except for the partition resizing.
I moved from a 64GB SSD to a 150GB VelociRaptor on Windows 7.

Instead of resizing the partition in Knoppix, I had to resize it within Windows, otherwise the partition would resize but the drive size in Windows Explorer would not. Quite strange!

Thanks for your article :D

John
28 Feb 2010, 07:09

Thanks for the tutorial, it was helpful. I had to do some more research though for the case when the disk geometry was different, and for a different partition layout. That requires some hacking of the partition boot sector and the registry to get windows to boot.

This page has some details on this: http://www.2pi.info/software/copying-windows-new-hard-drive.html

NEWS CONTENTS

• 20110726 : Mounting disk images on Linux loopback device ( Mounting disk images on Linux loopback device, Jul 26, 2011 )
• 20110726 : Creating and using disk images mini-howto Marcs Realm ( Creating and using disk images mini-howto Marc's Realm, Jul 26, 2011 )
• 20100310 : mount_dd ( freshmeat.net )
• 20100309 : Copying Windows to a new drive, using linux - How-to-Guide by Ed Anderson ( 20 Mar 2005 )
• 20070214 : How and when to use the dd command ( linux-noob.com-forums )
• 20070214 : Network Security Blog Using DD to clone a HD ( Network Security Blog Using DD to clone a HD, )
• 20070214 : Thomas Rude - DD and Computer Forensics - Hes Worth a Deuce! ( Thomas Rude - DD and Computer Forensics - He's Worth a Deuce!, )

Old News ;-)

[Jul 26, 2011] Mounting disk images on Linux loopback device

Disk space. There's never enough. Whilst preping my Inspiron 3800 for its new 20GB Toshiba 4500 RPM disk I thought I'd play around some with disk imaging. Playing with partition images is boring, so let's spice it up!

Obtaining a Disk Image

To start, you will want an exact image of a disk; Preferably one with filesystems you have support available for in your kernel, but any will do. As always, dd is your friend.

To obtain my disk image, I simply issued:

 
 rachael:#  dd if=/dev/hda of=/mnt/nebula/hda_dd.image
 4757130+0 records in
 4757130+0 records out

You can't simply mount a disk with the loopback device, however. You need some additional information. You will want to fetch a copy of the partition table, including the all important cylinder number we will use later. Invoke the magic of fdisk:

 rachael:/home/jasonb#  fdisk -l
 Disk /dev/hda: 4871 MB, 4871301120 bytes
 255 heads, 63 sectors/track, 592 cylinders
 Units = cylinders of 16065 * 512 = 8225280 bytes
    Device Boot    Start       End    Blocks   Id  System
 /dev/hda1   * 1       463   3719016    7  HPFS/NTFS
 /dev/hda2   464       592   1036192+   5  Extended
 /dev/hda5   464       479    128488+  82  Linux swap
 /dev/hda6   480       592    907641   83  Linux

Later, you can use this information to verifiy your image is sane.

Verifying the Sanity of Your Image

fdisk is quite effective for this task, too. You will need the cylinder number you obtained earlier either from fdisk, as shown above, or via some other means. (The 'C' option to fdisk is relatively recent. v2.11z has it; v2.11n that shipped with RedHat 7.3 does not. You can specify this from within fdisk by loading the image and using the e'x'pert mode and specifying the 'c' option from there.)

 
 faith:/home/jasonb#  fdisk -C 592 /nebula/hda_dd.image
 Command (m for help): p
 Disk /nebula/hda_dd.image: 0 MB, 0 bytes
 255 heads, 63 sectors/track, 592 cylinders
 Units = cylinders of 16065 * 512 = 8225280 bytes
        Device Boot    Start       End    Blocks   Id  System
 /nebula/hda_dd.image1   * 1       463   3719016    7  HPFS/NTFS
 /nebula/hda_dd.image2   464       592   1036192+   5  Extended
 /nebula/hda_dd.image5   464       479    128488+  82  Linux swap
 /nebula/hda_dd.image6   480       592    907641   83  Linux
 

Looks familiar, no? If all went well, it should be identical to the image yanked from the original disk.

Accessing Specific Partitions in the Image

Today, you might want to use multipath-tools instead, which includes kpartx for easier handling. Read on if you're curious about the old fashion, more brittle way of doing things.

Now, the fun begins. There are three ways to mount partitions from the image. You can simply use the stock kernel's loopback device, an enhanced loopback device offered by NASA, or extract the partition from the image and mount that directly with the loopback device. In all instances, the loopback device is the final destination. The journey varies with each, however. Let's look at the former most approach first.

Mounting with a Specified offset

The simplest method, you mount the partition of your choice from within the image. You will need to specify an offset for the loopback device into the image file. You can obtain this number by running fdisk against the image to obtain the starting and ending sectors for each partition. (Again, the -C option is only available in very recent versions of fdisk, like 2.11z.)

 
 faith:/home/jasonb#  fdisk -l -u -C 592 /nebula/hda_dd.image
 Disk /nebula/hda_dd.image: 0 MB, 0 bytes
 255 heads, 63 sectors/track, 592 cylinders, total 0 sectors
 Units = sectors of 1 * 512 = 512 bytes
        Device Boot    Start       End    Blocks   Id  System
 /nebula/hda_dd.image1   *63   7438094   3719016    7  HPFS/NTFS
 /nebula/hda_dd.image2       7438095   9510479   1036192+   5  Extended
 /nebula/hda_dd.image5       7438158   7695134    128488+  82  Linux swap
 /nebula/hda_dd.image6       7695198   9510479    907641   83  Linux

The offset must be specified in bytes, so now you must take the starting offset, in this instance 63, and multiply it by 512 bytes. From this we obtain 32256. (This assumes 63 sectors per track and 512 bytes per sector.) The file system type in this case is NTFS, so let us mount this partition from within the image using the usual loopback method.

 
 faith:/usr/src#  mount -o loop,offset=32256 \
   -t ntfs /nebula/hda_dd.image /mnt
 faith:/usr/src#  ls /mnt
 AUTOEXEC.BAT
 boot.ini
 CONFIG.SYS
 Corel
 Documents and Settings
 IO.SYS
 MSDOS.SYS
 NTDETECT.COM
 ntldr
 PUTTY.RND
 Program Files
 pagefile.sys
 RECYCLER
 System Volume Information
 WINNT

If you are using util-linux prior to version 2.12b, specifying an offset that required more than 32-bits was not possible. If you have util-linux 2.12b or newer, you can safely skip the next few sections. (You may still wish to extract individual partitions from your disk image using dd discussed at the end of this guide.)

Attempting to mount my ext3 partition near the end of the disk with a 2.11 version of util-linux yields (7695198 * 512 = 3939941376):

 
 faith:/usr/src#  mount -o loop,offset=3939941376 \
   -t ext3 /nebula/hda_dd.image /mnt
 mount: wrong fs type, bad option, bad superblock on /dev/loop0,
        or too many mounted file systems 

Fortunately, we aren't done yet. The second method utilizes a loopback device designed to mount partitions within the image without an offset limitation. In fact, no offset need be specified at all.

... ... ...

Mounting by First Extracting the Partition

Last, you can use dd to extract the partition of interest manually and then mount it via loopback. Again, the assumption of 512 bytes per sector is assumed here. As explained in Brian Carrier's March 15th Sleuth Kit Informer column, Splitting The Disk, we can pass dd the starting sector of the partition in question and calculate the size and allow it to extract it for us. For example, let's extract my ext3 partition, then mount it on loopback.

We pass dd bytes at a time size (bs option) of 512. Next, we pass it the starting sector of my ext3 partition from the fdisk output above, 7695198, as the number of blocks to skip ahead in the image. Last, we calculate the size as explained in the Sleuth Kit Informer above by taking the starting and ending sectors of the partition, subtracting them, then adding one (9510479 - 7695198 + 1 = 1815282).

Ronald Woelfel raised an interesting question about a missing sector on partitions with an odd number of sectors, which was explained thusly by Brian Carrier of Sleuth Kit fame: "The reason that you noticing the difference is likely because your linux system has the 2.4 kernel, which has a bug when accessing disk or partition devices. If a partition or disk has an odd number of sectors, the last sector is not read."

 
 faith:/home/jasonb#  dd if=/nebula/hda_dd.image of=/nebula/test.image \
   bs=512 skip=7695198 count=1815282
 1815282+0 records in
 1815282+0 records out
 

Once dd completes, you can mount the image as you normally would:

 
 faith:/home/jasonb#  mount -o loop -t ext3 /nebula/test.image /mnt
 faith:/home/jasonb#  ls /mnt
 bin    dev     home    lib opt   sbin  var
 boot   etc     import  lost+found  proc  tmp   vmlinuz
 cdrom  floppy  initrd  mnt root  usr   vmlinuz.old
 faith:/home/jasonb#  umount /mnt
 

Enjoy!

Links and Useful Resources

Security Focus mailing list thread on dd and mounting disk images that inspired much of this article

How to use dd to extract individual partitions from a disk image

[Jul 26, 2011] Creating and using disk images mini-howto Marc's Realm

Table of contents [hide]

1. Introduction
2. Creating a disk image
3. Mounting the disk image
   1. The clean way
   2. The dirty way
4. Final comments
5. Credits

Introduction

I had the need of booting Linux from a compact flash card in an embedded device. And for producing small quantities in-house it was necessary to have disk images from these flash cards that could just be dd'ed onto the the flash cards. Since it took me some time to figure out how to do that I've written this guide in hope others may find it useful.

Our first method to get these disk images was to boot the embedded device via Knoppix, mount the compact flash, copy the necessary files via scp onto the compact flash, chroot onto the flash, run LiLo and reboot. Afterwards we'd dd the complete flash content into a file. Not very entertaining. And especially hard to automate when you have a new release very often.

So we wanted to automate the image creating process as much as possible. Using an USB CF reader/writer we thought this shouldn't be too hard, but it turned out that when we copied the files onto the flash and chroot'ed into it lilo refused to run (can't remember why, sorry).

So we got the idea of producing bootable mini-images, where we would mount the partition using the loopback device, copy the files in, unmount the image and dd that complete image onto the compact flash (complete with MBR, partition table, everything).

Problem is, again lilo is making problems: you can't just update the kernel by copying a new one over the old one. You have to run lilo again. And grub was out since neither of us managed to get it to work (while grub seems to be very good, the configuration is an unnecessarily hairy nightmare Almost the same technique as described in this mini-howto can be use with grub if you dig it. And saves you the FAT16 partition as well.). Alternatives: booting DOS, using LOADLIN or SYSLINUX. Obviously, SYSLINUX is the cleaner solution.

This simple technique described can also be used with any other medium, like USB sticks for example.

Creating a disk image

And this is how you do it:

1. Insert CF into reader/writer. We assume that the CF is now accessible as /dev/sda.
2. Since our raw CF's had lots of garbage on it we zero out the complete CF (helps compressing the image later on We used 64MB CF's when I wrote the first version of this documents. When zero'd, partitioned and formated these compressed down to just 4200 bytes with bzip2... nice ratio :-) An even nicer ratio is that of a 2GB hard disk image we've done: it compresses from 2GB down to just 18613 bytes.).

   dd if=/dev/zero of=/dev/sda

3. Create partitions: we need at least one boot partition (FAT12 or FAT16, but not FAT32) and a root partition (we used Ext3).

   fdisk /dev/sda

4. Format the partitions.

   mkfs.msdos /dev/sda1
   mkfs.ext3 /dev/sda2

5. Install SYSLINUX on boot partition.

   syslinux -s /dev/sda1

6. Install master boot record (found in SYSLINUX source directory).

   dd if=mbr.bin of=/dev/sda

7. Mount the boot partition.

   mount /dev/sda1 /mnt

8. Copy the kernel image onto boot partition.

   cp bzImage /mnt/kernel.bzi

9. Create SYSLINUX configuration file.

   cat >/mnt/syslinux.cfg <<"EOF"
   DEFAULT kernel

   LABEL kernel
   KERNEL kernel.bzi
   APPEND root=/dev/hdc2
   EOF

10. Umount the boot partition.

    umount /mnt

11. Save the final image.

    dd if=/dev/sda of=image.bootable

You can then mount the root directory and copy all your files into it, and even update the kernel by just copying a new bzImage onto the boot partition. No need to run any program like LiLo afterwards.

If you just want to copy the partitioned space then you may want to read on about mounting the disk image and then come back here: you need to calculate the size, which is (<end block number of the last partition> + 1) * 512. Then give dd the additional option count=<size>.

Mounting the disk image

There are two ways to mount the partition.

The clean way

First, we need to determine the offset of the partition. This is quite easy: just type fdisk -ul <device>. The option -ul means list the partitions on the device and assume a unit size of 512 byte. This looks something like this:

tetsuo:~ # fdisk -ul /dev/sda

Disk /dev/sda: 256 MB, 256376832 bytes
8 heads, 62 sectors/track, 1009 cylinders, total 500736 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 62 19839 9889 4 FAT16 <32M
/dev/sda2 19840 231135 105648 83 Linux
/dev/sda3 231136 442431 105648 83 Linux
/dev/sda4 442432 471199 14384 83 Linux

Now all we need to do is a little math to get the offset: we need to multiply the start block by 512. E.g. if we wanted to mount the first partition we'd have an offset of 62 * 512 = 31744. The second partition has an offset of 19840 * 512 = 10158080. Now that we have the offset we can mount the partition:

mount -o loop,offset=10158080 image.bootable /mnt

This would mount the second partition on /mnt. Linux recognizes it as ext3 if it is formatted as ext3 and the kernel supports ext3, so no need for a -t ext3 option to mount.

The dirty way

There is also a hard way to find the formatted partitions if you can't calculate the offsets for some reason:

for ((i=0 ; $i < 10000 ; i=$i + 1)) ; do
mount -o loop,offset=$(($i * 512)) image.bootable /mnt && break
done

If there is a partition within the first 10000 blocks, it gets mounted eventually :-) Just type "mount" to get the offset...

Final comments

After we've unmounted the disk image we can now just dd the disk image to a new compact flash:

dd if=image.bootable of=/dev/sda

Easy as that.

There are several ways to force Linux to re-read the partition table after we've written a disk image with partition table to an empty compact flash. Propably the best way is to run:

partprobe

This program is part of GNU parted. If it's not installed then you might succeed with the following command:

/sbin/sfdisk -R /dev/sda

In the rare case that you have neither, there's still a hack: unload the USB module and load it again:

modprobe -r usb-uhci && modprobe usb-uhci

Credits

This document was written and is ©opyrighted 2003,2006,2010 by Marc Haisenko. Thanks to the SYSLINUX author H. Peter Anvin for finding an unnecessary step in the creation process. This moved to the chapter "Final comments". Manicalic told me about partprobe and sfdisk to re-read the partition table. If you have further comments/additions/corrections please mail them to me. You may copy and distribute this document as long as you include this credit section and my name. You may modify it and add your name to this section as well.

[Mar 10, 2010] mount_dd

freshmeat.net

mount_dd is a small tool for mounting a dd image with a GUI. You can mount it in read-write or read-only mode.

[Mar 9, 2010] Copying Windows to a new drive, using linux - How-to-Guide by Ed Anderson

20 Mar 2005

Updated 18 May 2009 (NTFS Updates; device names)

This guide will show you how to copy an existing installation of Windows (or any other OS) from one drive to another - as long as the destination drive is the same size or larger.

This is a free and relatively easy method that will create a clone of your current hard disk, without having to buy any third party software.

1. Gathering tools
2. Physical Installation
3. Preparing new partition table
4. Copy the MBR
5. Copy the Partition
6. Resizing the Partition
7. FAQ

[Feb 14, 2007] How and when to use the dd command

linux-noob.com-forums

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image 

1+0 records in
1+0 records out 

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image	

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy

dd bs=2x80x18b < /tmp/floppy.image 
	> /dev/fd0 

	1+0 records in 

	1+0 records out 

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

Example 2

The original need for dd came with the 1/2" tapes used to exchange data with other systems and boot and install Unix on the PDP/11. Those days are gone, but the 9-track format lives. To access the venerable 9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking and unblocking are no longer a necessity, as the hardware reads and writes 512-byte data blocks.

However, the 9-track 1/2" tape format allows for variable length blocking and can be impossible to read with the cp command. The dd command allows for the exact specification of input and output block sizes, and can even read variable length block sizes, by specifying an input buffer size larger than any of the blocks on the tape. Short blocks are read, and dd happily copies those to the output file without complaint, simply reporting on the number of complete and short blocks encountered.

Then there are the EBCDIC datasets transferred from such systems as MVS, which are almost always 80-character blank-padded Hollerith Card Images! No problem for dd, which will convert these to newline-terminated variable record length ASCII. Making the format is just as easy and dd again is the right tool for the job.

Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length newline-terminated record

dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out

40+0 records in
38+1 records out 

The fixed record length is specified by the cbs=80 parameter, and the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock parameter.

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

Example 3

Sometimes data arrives from sources in unusual formats. For example, every time I read a tape made on an SGI machine, the bytes are swapped. The dd command takes this in stride, swapping the bytes as required. The ability to use dd in a pipe with rsh means that the tape device on any *nix system is accessible, given the proper rlogin setup.

Example 3 : Byte Swapping with Remote Access of Magnet Tape

rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf - 

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

Example 4

Murphy's Law was postulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, it is the only copy in the universe and you have a deadline past due, that is when you will have a bad spot on the magnetic media, and your data will be unreadable. To the rescue comes dd, which can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.

Example 4 : Error Handling

dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image 

Example 5

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile

dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1 

This skips the first 512 bytes of the input bootimage file (skip=1) and writes starting at the second sector of the $(BOOTDEV) device (seek=1). A typical use of dd is to skip executable headers and begin writing in the middle of a device, skipping volume and partition data. As this can cause your disk to lose file system data, please test and use these applications with care.BigAdmin Submitted Article Backup and Recovery Utilities for the Solaris OS

dd -- Convert and Copy a File

The dd command is most commonly used to copy a complete file system to another file system or to copy a hard disk drive to another disk drive. dd can also be used to copy a file system to tape, and vice versa. dd is a relatively quick copy tool: It creates an exact copy (byte for byte) as it transfers the data. Several options are used with dd to specify buffer sizes, block sizes, and data conversions. The basic syntax for dd is:

dd <inputfile>= <outputfile>= options

The following example copies the entire contents of c0t1d0s2 to a second disk, c0t4d0s2, using a block size of 128. This works great if you have a spare disk available and want to have a backup disk ready to swap out in case of a disaster. Slice 2 is specified in this example because it represents the entire disk in the Solaris OS.

% dd if=/dev/rdsk/c0t1d0s2 of=/dev/rdsk/c0t4d0s2 bs=128

To copy the contents of one tape device to another, use the following example. (Note: You must have two drives available.)

% dd if=/dev/rmt/#1drive of=/dev/rmt/#2drive

The man pages on dd give many other options available for use. I suggest viewing the man pages for dd and also reading the Notes section. There you will find some limitations to dd, as well as a reminder to use the raw character device when copying data from disk devices.

Network Security Blog Using DD to clone a HD

Thomas Rude - DD and Computer Forensics - He's Worth a Deuce!

For quick review, let's create an image based on the following:
We have a 2GB hard disk we want to image. We connect the 2GB disk as a slave in our PC. Upon boot, we mount the 2GB disk read only. We are now ready to image! Using DD, how 'bout:

dd if=/dev/hdb of=/images/hdbApril42001.img

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019