Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

DD Command

News Recommended Links Recommended articles Reference Examples Recovery of lost files using DD Working with DD Images
Loopback filesystem dd and netcat Mount a partition from dd disk image DD Rescue Acronis True Image Ghost Alternatives to Norton Ghost
Filesystems Working with ISO Images Remote Backup Disk Repartitioning Sysadmin Horror Stories Humor Etc

Introduction

The dd command has been around since the 1970s, ported to many systems, rewritten many times. It proved to be an indispensable Unix tool. The name is an allusion to IBM/360 mainframe OS JCL DD statement. It is jokingly said that dd stands for "destroy disk" or "delete data", since, being used for low-level operations on hard disks, a small mistake, such as reversing the if and of parameters, may accidentally render the entire disk unusable. In modern Linuxes you can mount dd image using loopback interface.  Recently capability to mount dd-images was added to Linux and other Unixes.

DD is very important tool for recovery of Unix filesystem. The simpler is layout of your disk the more chances you have to recover information.  Complex setups involving LVM are more difficult to recover.

DD also can be use for recovery of files on Windows filesystems, especially FAT32. See Recovery of lost files using DD

The GNU clone of dd is part of fileutils package and was written by Paul Rubin, David MacKenzie, and Stuart Kemp. dd is also available for Windows as part of Microsoft Unix toolkit (SFU 3.5). It is also part of Cygwin. There is also native Windows Win32 port of dd (see Native Win32 ports of some GNU utilities and dd for windows)

It served as an inspiration to the most important recent class backup programs called ghosters in memory of the original for Windows Ghost. The key idea is to get an image of the partition in the form of the file. DD uses older approach and includes into image parts of the disk that are not parts of the filesystem. That feature can be used to recovery of deleted files and in computer forensics when the contents of a disk need to be preserved as a byte-exact copy. In the latter case using cp command would not be sufficient because data from deleted files still physically present on a disk but are not visible in the file system naming space.

In Windows dd command can do backups and restores to the same disk, but there are some problems with cloning Windows systems. Ghost and Acronis True Image are better tools for that. As a backup tool dd is still useful, but Ghost and its alternatives are much faster because they understand what part of the disk belongs to the filesystem and which don't. Utility dd copies the whole partition or the whole disk indiscriminately which make the image larger, although it contains deleted files and can be used for recovery of those files instead of the disk.

On Linux Partimage is analog of Gnome and is usable with ext filesystem up to ext3. Not usable with ext4.  Partition Image saves partitions or whole disks for the most common filesystems formats (Ext2, Ext3, NTFS, FAT32, etc) to an  image file. The image file can  further be compressed in the GZIP/BZIP2 formats to save disk space, and split into multiple files. Partitions can be saved across the network since version 0.6.0.When using Partimage, the partitions must be unmounted.

Recovery of lost files

That fact that deleted files are still present on the disk represents a serious (and decisive for data recovery and forensics) dd advantages over Ghost and similar, more modern, tools. In case of damaged filesystem dd is much more useful. You just need to create a DD-image and then search it for files using grep or similar tool. In most cases you can recover files pretty reliably using this method. I one manage to recover a script that was deleted several hours ago before it was discovered and server continued working all this time. That was the only copy and the user who did it has no backup.

It is jokingly said that dd stands for "destroy disk" or "delete data", since, being used for low-level operations on hard disks, a small mistake, such as reversing the if and of parameters, may accidentally render the entire disk unusable.

On linux you can mount dd image using "loopback" interface and work with it like a partition. See Recovery of lost files using DD

Syntax

Unlike most Unix commands, dd uses a keyword=value format for its parameters. This was modeled after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition' specification for I/O devices in JCL language.

The following exit values are returned:

After completion, dd reports the number of whole and partial input and output blocks

A complete listing of all keywords is available via dd --help. It depends on flavor of dd used: GNU dd is different from, say, Solaris dd. See Reference

Basic dd Options

The basic syntax of dd is as follows:

# dd \
    if=devicein \ # devide from which disk block are read
    of=device_of_file_out \ # device or file to which disk block are read
    bs=blocksize # blocksize \
    count=n  # number of block to copy

TIP: Looks like dd  works slightly faster with 128K blocks. 

The preceding options are used almost every time you run dd:

  1. if= file Specifies the input path. Standard input is the default. The if= argument specifies the input file or the file from which it is going to copy the data. This is the file or raw partition that you are going to back up (e.g., dd if=/dev/dsk/c0t0d0s0 or dd if=/home/file). If you want dd to look at stdin for its data, you don't need this argument.
  2. of= file Specifies the output path. Standard output is the default. If the seek=expr conversion is not also specified, the output file will be truncated before the copy begins, unless conv=notrunc is specified. If seek=expr is specified, but conv=notrunc is not, the effect of the copy will be to preserve the blocks in the output file over which dd seeks, but no other portion of the output file will be preserved. (If the size of the seek plus the size of the input file is less than the previous size of the output file, the output file is shortened by the copy.) The output file The of= argument specifies the output file or the file to which you are sending the data. This could be a file on disk or an optical platter, another raw partition (e.g., dd of=/backup/file, dd of=/dev/rmt/0n). If you are sending to stdout, you don't need this argument. the block size

    The bs= argument specifies the block size, or the amount of data that is to be transferred in one I/O operation. This value normally is expressed in bytes, but in most versions dd also can be specified in kilobytes by adding a k at the end of the number (e.g., 10 K). (This is different from a blocking factor, like dump and tar use, which is multiplied by a fixed value known as the minimum block size. A blocking factor of 20 with a minimum block size of 512 would give you an actual block size of 10,240, or 10 K.) It should be noted that when reading from or writing to a pipe, dd defaults to a block size of 1. Generally on modern hardware bs=4096 is a reasonable minimum value. One cylinder (255 heards * 63 sectors * 512 = 16065. So bs=16065 also is attractive. Two cylinders fro multiplate disks (bs=32130) also might make sense.

    Changing block size does not affect how the data is physically written to a disk device, such as a file on disk or optical platter. Using a large block size just makes the data transfer more efficient. When writing to a tape device, however, each block becomes a record, and each record is separated by an interrecord gap. Once a tape is written with a certain block size, it must be read with that block size or a multiple of that block size. (For example, if a tape were written with a block size of 1024, you must use the block size of 1024 when reading it, or you may use 2048 or 10,240, which are multiples of 1024.) Again, this applies only to tape devices, not disk-like devices.
     

  3. bs= n Sets both input and output block sizes to n bytes, superseding ibs= and obs=. If no conversion other than sync, noerror, and notrunc is specified, each input block is copied to the output as a single block without aggregating short blocks. This option is used only if ASCII or EBCDIC conversion is specified. For the ascii and asciib operands, the input is handled as described for the unblock operand except that characters are converted to ASCII before the trailing SPACE characters are deleted. For the ebcdic, ebcdicb, ibm, and ibmb operands, the input is handled as described for the block operand except that the characters are converted to EBCDIC or IBM EBCDIC after the trailing SPACE characters are added. Specifying the input and output block sizes separately When specifying block size with the option bs=, you are specifying both the incoming and outgoing block size. There are situations in which you may need a different block size on each. This is done with the ibs= and obs= options.
     
  4. count=n the number of records to read The count=n option tells dd how many records (blocks) to read. You can use this to read the first few blocks of a file or tape to see what kind of data it is, for example (see the following section). You can also use it to have dd tell you what block size a tape was written in (see below).

The dd utility copies the specified input file to the specified output with possible conversions. If you reverse the source and target, you can wipe out you file, partitions or even the whole disk. This feature has inspired the nickname "dd" Data Destroyer.

The standard input and output are used by default. The input and output block sizes may be specified for tape or just to increase efficiency (large blocks are generally transferred faster). Sizes are specified in bytes; a number may end with k, b, or w to specify multiplication by 1024, 512, or 2, respectively. Numbers may also be separated by x to indicate multiplication.

The dd utility reads the input one block at a time, using the specified input block size. dd then processes the block of data actually returned, which could be smaller than the requested block size. dd applies any conversions that have been specified and writes the resulting data to the output in blocks of the specified output block size.

Additional options

cbs= n
Specifies the conversion block size for block and unblock in bytes by n (default is 0). If cbs= is omitted or given a value of 0, using block or unblock produces unspecified results. bs is used only if ascii, asciib, unblock, ebcdic, ebcdicb, ibm, ibmb, or block conversion is specified. In the first two cases, cbs characters are copied into the conversion buffer, any specified character mapping is done, trailing blanks are trimmed, and a NEWLINE is added before sending the line to output. In the last three cases, characters up to NEWLINE are read into the conversion buffer and blanks are added to make up an output record of size cbs. ASCII files are presumed to contain NEWLINE characters. If cbs is unspecified or 0, the ascii, asciib, ebcdic, ebcdicb, ibm, and ibmb options convert the character set without changing the input file's block structure. The unblock and block options become a simple file copy.
files= n
Copies and concatenates n input files before terminating (makes sense only where input is a magnetic tape or similar device).
skip= n
Skips n input blocks (using the specified input block size) before starting to copy. On seekable files, the implementation reads the blocks or seeks past them. On non-seekable files, the blocks are read and the data is discarded.
iseek= n
Seeks n blocks from beginning of input file before copying (appropriate for disk files, where skip can be incredibly slow).
oseek= n
Seeks n blocks from beginning of output file before copying.
seek= n
Skips n blocks (using the specified output block size) from beginning of output file before copying. On non-seekable files, existing blocks are read and space from the current end-of-file to the specified offset, if any, is filled with null bytes. On seekable files, the implementation seeks to the specified offset or reads the blocks as described for non-seekable files.
count= n
Copies only n input blocks.
conv= value[,value. . . ]
Where values are comma-separated symbols from the following list:
ascii
Converts EBCDIC to ASCII.
asciib
Converts EBCDIC to ASCII using BSD-compatible character translations.
ebcdic
Converts ASCII to EBCDIC. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.
ebcdicb
Converts ASCII to EBCDIC using BSD-compatible character translations. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.
ibm
Slightly different map of ASCII to EBCDIC. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.
ibmb
Slightly different map of ASCII to EBCDIC using BSD-compatible character translations. If converting fixed-length ASCII records without NEWLINEs, sets up a pipeline with dd conv=unblock beforehand.

The ascii (or asciib), ebcdic (or ebcdicb), and ibm (or ibmb) values are mutually exclusive.

block
Treats the input as a sequence of NEWLINE-terminated or EOF-terminated variable-length records independent of the input block boundaries. Each record is converted to a record with a fixed length specified by the conversion block size. Any NEWLINE character is removed from the input line. SPACE characters are appended to lines that are shorter than their conversion block size to fill the block. Lines that are longer than the conversion block size are truncated to the largest number of characters that will fit into that size. The number of truncated lines is reported.
unblock
Converts fixed-length records to variable length. Reads a number of bytes equal to the conversion block size (or the number of bytes remaining in the input, if less than the conversion block size), delete all trailing SPACE characters, and append a NEWLINE character.

The block and unblock values are mutually exclusive.

lcase
Maps upper-case characters specified by the LC_CTYPE keyword tolower to the corresponding lower-case character. Characters for which no mapping is specified are not modified by this conversion.
ucase
Maps lower-case characters specified by the LC_CTYPE keyword toupper to the corresponding upper-case character. Characters for which no mapping is specified are not modified by this conversion.

The lcase and ucase symbols are mutually exclusive.

swab
Swaps every pair of input bytes. If the current input record is an odd number of bytes, the last byte in the input record is ignored.
noerror
Does not stop processing on an input error. When an input error occurs, a diagnostic message is written on standard error, followed by the current input and output block counts in the same format as used at completion. If the sync conversion is specified, the missing input is replaced with null bytes and processed normally. Otherwise, the input block will be omitted from the output.
notrunc
Does not truncate the output file. Preserves blocks in the output file not explicitly written by this invocation of dd. (See also the preceding of=file operand.)
sync
Pads every input block to the size of the ibs= buffer, appending null bytes. (If either block or unblock is also specified, appends SPACE characters, rather than null bytes.)

If operands other than conv= are specified more than once, the last specified operand=value is used.

For the bs=, cbs=, ibs=, and obs= operands, the application must supply an expression specifying a size in bytes. The expression, expr, can be:

  1. a positive decimal number
  2. a positive decimal number followed by k, specifying multiplication by 1024
  3. a positive decimal number followed by b, specifying multiplication by 512
  4. two or more positive decimal numbers (with or without k or b) separated by x, specifying the product of the indicated values.

All of the operands will be processed before any input is read.

Although you may think of dd as a bit copier, it also can manipulate the format of the data, such as converting between different character sets, upper- and lowercase, and fixed-length and variable-length records.

conv=ascii
Converts EBCDIC to ASCII
conv=ebcdic
Converts ASCII to EBCDIC
conv=ibm
Converts ASCII to EBCDIC using the IBM conversion table
conv=lcase
Maps US ASCII alphabetic characters to their lowercase counterparts
conv=ucase
Maps US ASCII alphabetic characters to their uppercase counterparts
conv=swab
Swaps every pair of bytes; can be used to read a volume written in different byte order
conv=noerror
Does not stop processing on an error
conv=sync
Pads every input block to input block size (ibs)
conv=notrunc
Does not truncate existing file on output
conv=block
Converts input record to a fixed length specified by cbs
conv=unblock
Converts fixed-length records to variable length
conv=..., ...
Uses multiple conversion methods separated by commas

Examples

NOTES
Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Aug 24, 2014] 5 Awesome Open Source Cloning Software

dcfldd: A fork of dd

dcfldd is an enhanced version of GNU dd with features useful for forensics and security. Here is an example of cloning a hard disk "sda" and store to an image called "/nfs/sda-image-server2.dd":

 
dcfldd if=/dev/sda hash=md5,sha256 hashwindow=10G md5log=md5.txt sha256log=sha256.txt \
       hashconv=after bs=512 conv=noerror,sync split=10G splitformat=aa of=/nfs/sda-image-server2.dd
 

GNU ddrescue is a data recovery tool. It copies data from one file or block device (hard disc, cdrom, etc) to another, trying to rescue the good parts first in case of read errors.

=> Download dcfldd and GNU dd (GNU core utilities and installed on most Unix-like systems)

[May 08, 2014] 11 Awesome DD Commands

Jan 12, 2011 | UrFix's Blog
1) Duplicate several drives concurrently
dd if=/dev/sda | tee >(dd of=/dev/sdb) | dd of=/dev/sdc

If you have some drive imaging to do, you can boot into any liveCD and use a commodity machine. The drives will be written in parallel.

To improve efficiency, specify a larger block size in dd:

dd if=/dev/sda bs=64k | tee >(dd of=/dev/sdb bs=64k) | dd of=/dev/sdc bs=64kTo image more drives , insert them as additional arguments to tee:

dd if=/dev/sda | tee >(dd of=/dev/sdb) >(dd of=/dev/sdc) >(dd of=/dev/sdd) | dd of=/dev/sde

2) create an emergency swapfile when the existing swap space is getting tight

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024000
sudo mkswap /swapfile; sudo swapon /swapfile

Create a temporary file that acts as swap space. In this example it's a 1GB file at the root of the file system. This additional capacity is added to the existing swap space.

3) Backup your hard drive with dd
sudo dd if=/dev/sda of=/media/disk/backup/sda.backup

This will create an exact duplicate image of your hard drive that you can then restore by simply reversing the "if" & "of" locations.

sudo dd if=/media/disk/backup/sda.backup of=/dev/sdaAlternatively, you can use an SSH connection to do your backups:

dd if=/dev/sda | ssh [email protected] dd of=~/backup/sda.backup

4) Convert a Nero Image File to ISO

dd bs=1k if=image.nrg of=image.iso skip=300

This line removes the 300k header from a Nero image file converting it to ISO format

5) send DD a signal to print its progress

while :;do killall -USR1 dd;sleep 1;done

every 1sec sends DD the USR1 signal which causes DD to print its progress.

6) show dd progress part 2

killall -USR1 dd

if you need see progress of long dd command, enter subj on other console

7) How to copy CD/DVD into hard disk (.iso)

dd if=/dev/cdrom of=whatever.iso

A dear friend of mine asked me how do I copy a DVD to your hard drive? If you want to make a copy of the ISO image that was burned to a CD or DVD, insert that medium into your CD/DVD drive and (assuming /dev/cdrom is associated with your computer?s CD drive) type the following command

8) Watch the progress of 'dd'

dd if=/dev/zero | pv | dd of=/dev/null

need pv (pipe view) :

http://www.ivarch.com/programs/pv.shtml

9) Clone IDE Hard Disk

sudo dd if=/dev/hda1 of=/dev/hdb2

This command clone the first partition of the primary master IDE drive to the second partition

of the primary slave IDE drive (!!! back up all data before trying anything like this !!!)

10) Test network speed without wasting disk
dd if=/dev/zero bs=4096 count=1048576 | ssh [email protected] 'cat > /dev/null'

The above command will send 4GB of data from one host to the next over the network, without consuming any unnecessary disk on either the client nor the host. This is a quick and dirty way to benchmark network speed without wasting any time or disk space.

Of course, change the byte size and count as necessary.

This command also doesn't rely on any extra 3rd party utilities, as dd, ssh, cat, /dev/zero and /dev/null are installed on all major Unix-like operating systems.

11) clone a hard drive to a remote directory via ssh tunnel, and compressing the image
dd if=/dev/sda | gzip -c | ssh user@ip 'dd of=/mnt/backups/sda.dd'

[Dec 04, 2013] Backup-Recovery - Cloning Hard Drives with GNU-Linux

2002-12-25 | netadmintools.com

It is quite easy to clone identical hard drives using the dd command on GNU/Linux. Make sure that you put the source drive and destination drive in the system so that they don't affect the boot. If you have a SCSI system, this is most likely done by making the SCSI IDs higher. With IDE, you probably need to put the drives in as secondaries on either channel, assuming your CD-ROM is the primary device on the second chain. On our system, we made the source ID 4 and the destination ID 6. Verify this using dmesg:

[root@srv-33 root]# dmesg | grep sd
Kernel command line: ro root=/dev/sda1 console=ttyS0 
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi0, channel 0, id 4, lun 0
Attached scsi disk sdc at scsi0, channel 0, id 6, lun 0
SCSI device sda: 2069860 512-byte hdwr sectors (1060 MB)
sda: sda1 sda2
SCSI device sdb: 8388315 512-byte hdwr sectors (4295 MB)
sdb: sdb1 sdb2
SCSI device sdc: 8388315 512-byte hdwr sectors (4295 MB)
sdc: unknown partition table
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal
[root@srv-33 root]# 
Notice that with this copy method, we don't need to worry about partitions, boot sectors, etc. To copy sdb to sdc:
[root@srv-33 root]# dd if=/dev/sdb of=/dev/sdc &
[1] 612
[root@srv-33 root]# 
The ampersand throws the task in the background. You may then copy to other destination drives if you wish. Again, this copies everything on the drive (REALLY!!), so you can image any operating system. One cheapie way to recover a server is to image the hard drive in this way. The SIDs (Windows) will be identical as well. You will be at the exact same state as the time of imaging. No worries about open files, as long as you shut down the OS properly on the drive first. This method is not suitable for cloning Windows (NT and above) workstations that will be up at the same time, unless you feel comfortable changing the SIDs, etc. Test this first in a non-production environment, and, oh yeah, read our terms of use.

[Jan 05, 2012] Scalpel: A Frugal, High Performance File Carver

digitalforensicssolutions

Scalpel is a fast file carver that reads a database of header and footer definitions and extracts matching files or data fragments from a set of image files or raw device files. Scalpel is filesystem-independent and will carve files from FATx, NTFS, ext2/3, HFS+, or raw partitions. It is useful for both digital forensics investigation and file recovery.

Notes on Platforms

Linux

The preferred platform for using Scalpel is Linux.

Windows

Scalpel will also compile under Windows (32 or 64-bit) using mingw. If you'd like to try Scalpel on Windows without the bother of compiling it yourself, an executable and appropriate libraries are included in the distribution--just untar and go. Note that under Windows, the pthreads DLL must be present in the same directory as the Scalpel executable. Carving physical and logical devices directly under Windows (e.g., using \\.\physicaldrive0 as a target) is not supported in the current release.

Mac OS X

As of v1.53, Scalpel is supported on Mac OS X.

All platforms

As of v1.54, Scalpel supports carving files larger than 4GB on all platforms.

As of v1.60, Scalpel supports preview carving and other new carving modes. See the distribution for details.

As for v2.0, Scalpel supports regular expressions for headers and footers, minimum carve sizes, multithreading and asynchronous I/O, and beta-level support for GPU-accelerated file carving.

[Jan 19, 2011] Image software

LinkedIn

Leon Waldman

I always bet on dd + netcat to do imaging over network.

On the system where you will store the image (192.168.0.10):
netcat –l –p 7000 > file.iso

On the system to be cloned:
dd if=/dev/sda | netcat 192.168.0.10 7000 -q
where: dd if=volume_to_be_cloned| netcat ip_of_destination_ip port -q

You can even do it to full clone the system instead store it.

Give a look here:
http://digiassn.blogspot.com/2006/01/dd-over-netcat-for-cheap-ghost.html

[Mar 9, 2010] Copying Windows to a new drive, using linux - How-to-Guide by Ed Anderson

20 Mar 2005 | nilbus.com

Updated 18 May 2009 (NTFS Updates; device names)

This guide will show you how to copy an existing installation of Windows (or any other OS) from one drive to another - as long as the destination drive is the same size or larger.

This is a free and relatively easy method that will create a clone of your current hard disk, without having to buy any third party software.

  1. Gathering tools
  2. Physical Installation
  3. Preparing new partition table
  4. Copy the MBR
  5. Copy the Partition
  6. Resizing the Partition
  7. FAQ

[Feb 11, 2009] Create a file with given size - Linux dd command

Feb 11, 2009 |unstableme.blogspot.com

"The command is:

dd if=/dev/zero of=testfile_10MB bs=10485760 count=1

	"1+0 records in
	1+0 records out
10485760 bytes (10 MB) copied, 0.312 s, 33.6 MB/s"

[Oct 3, 2008] Red Hat Magazine This isn't your grandpappy's dd command

Looks like dd was slightly faster with 128K blocks. Python program might be reused for other purposes
Oct 02, 2008 | redhatmagazine.com
Block Size: 128 Throughput: 62.8 MB/s
Block Size: 256 Throughput: 61.8 MB/s
Block Size: 512 Throughput: 57.1 MB/s
Block Size: 1024 Throughput: 56.5 MB/s
We benchmarked the throughput of the disk by running the dd command with various block sizes from 128 KB to 1 MB. (Note: If you want to run the script on your own machine, make sure that the volume you use doesn't contain any valuable data, because the data will be erased by the dd command. Remember, data loss makes grandpappy mad!)

For the benchmark, we wrote a Python script that uses the commands module to run and capture the output of the dd command. The script also uses the csv module to generate a comma-separated values file so that we can graph the results later. For this example, we chose to graph the results using the Google Chart API.

dd_chart on Flickr - Photo Sharing!

[Sep 12, 2008] Tips For Linux - How and when to use the dd command

[Oct 1, 2004 ] POWER TOOLS Performing Data Surgery

Data Dumping with dd
October 1, 2004 | Linux Magazine

dd does low-level data transfer, byte-by-byte or block-by-block, with adjustable block sizes. It can also skip specified numbers of blocks in the input and/or output files, as well as converting data formats. All of those are handy for working with magnetic tape and disks. But it's also useful for many types of data transfers.

By default, dd reads the standard input and writes to the standard output. Input and output filenames, and other options too, are given in an unusual syntax without leading dash ( - ) characters.

For instance, to read a floppy disk and write its image to a file, you could type:

$ dd if=/dev/fd0 of=dosboot.img
2880+0 records in
2880+0 records out
$ ls -l dosboot.img
-rw-rw-r- ... 1474560 Nov 2 12:59 dosboot.img

The dd command line says, "Reading from the input file /dev/fd0, write all of the data to the file dosboot.img." dd doesn't try to find lines of data or individual files on the disk; it does a binary copy of the bytes from first to last. dd always tells you (on the standard error) how many times it read and wrote data. Above, it read 2,880 512-byte blocks. If you don't want to see this information -- or any error messages, either -- you can redirect dd's standard error to the Linux "bit bucket," /dev/null, by adding the Bourne shell operator 2>/dev/ null to the command line.

It's more efficient to specify a larger block size so the device drivers do a single read and write. There are lots of other options, and many of them start with conv= , like conv=unblock to replace trailing spaces in a block with a newline, and conv= swap to swap pairs of input bytes (which is needed with some tapes written on other types of hardware). But we'll leave that sort of optimization to you and the dd man page. Let's look at some less-obvious uses of this handy utility.

Stupid dd Tricks

Need a file with 100 arbitrary bytes -- for testing, for instance? The Linux device /dev/urandom (available since Linux 1.3.30) can supply as many pseudo-random bytes as you can read from it. To get just 100 bytes, set a block size of 1 byte with bs=1 and tell dd to stop after copying 100 "blocks" (here, that's 100 bytes):

$ dd if=/dev/urandom of=myrand bs=1 count=100

What's in that myrand file? The od utility can show you. (See the sidebar "What's In That File?")

If you need more-random data, try /dev/random instead. Reading data from /dev/random can take some time, though, as the random(4) man page explains. When you read from /dev /random, set a block size of 1.

Another use for dd is for "wiping" a text file before you delete it. Simply removing a Linux file (with rm, for instance) only deletes the inode that points to the data. A cracker with root access might read the raw disk (with dd!) and find the "deleted" file. We can use dd to write random data over the file before deleting it. Normally dd truncates a file before writing, so use conv=notrunc to make it write over the existing data. Set bs to the file size and count to 1 . For example:

% ls -l afile
-rw------- ... 3769 Nov  2 13:41 afile
% dd if=/dev/urandom of=afile \
  bs=3769 count=1 conv=notrunc
1+0 records in
1+0 records out
% rm afile

If you want to, you can repeat the "wiping" command several times with the C shell repeat command, the Z shell repeat loop, or simply use the history operator !!

[Oct 1, 2004 ] Moving your data to a backup device

faqs.org

The dd command can be used to put data on a disk, or get it off again, depending on the given input and output devices. An example:

gaby:~>dd if=images-without-dir.tar.gz of=/dev/fd0H1440
98+1 records in
98+1 records out

gaby~>dd if=/dev/fd0H1440 of=/var/tmp/images.tar.gz
2880+0 records in
2880+0 records out

gaby:~>ls /var/tmp/images*
/var/tmp/images.tar.gz

Note that the dumping is done on an unmounted device. Floppies created using this method will not be mountable in the file system, but it is of course the way to go for creating boot or rescue disks. For more information on the possibilities of dd, read the man pages.

This tool is part of the GNU fileutils package.

[Feb 13, 1998] SUMMARY How to build a Solaris bootable CDROM by James Hutchinson

sunmanagers.org

One would think this would be a simple task. Half the commerical packages proclaim that they can do it. But alas, for me, none of them worked.

The task was to create a system installer that would boot a sun, load up the appropriate system images and tools required for a server in our environment. Yes, this is a perfect job for jumpstart, but the systems would be distributed across the state and not connected via a fast enough pipe to use jumpstart effectivly. The other option was to build external disks with the proper images on them and boot from there. The cost of producing a few hundrad of these was prohibitive, thus the CDROM approach was taken.

Of the few responses I received from Sun Managers readers, all of them basicly said 'Its easy to clone the install disc!' or 'it just cant be done!'

Well. It is possible.

First, a breakdown of the process. When the command > boot cdrom < is givin to openboot the system actually looks at slice 2 to 5 for its bootblks depending on the machine type. A sun4c is slice 2 to an sun4u at slice 5. These bootblks redirect the system to slice 1 to load its kernel. The root partition is also stored in slice 1 and slice 0 is usr as well as packages. All slices save 0 are ufs, and 0 is hsfs.

This brings us to a small problem point. First we must boot from a UFS partition, as the bootblks all require that. Second, a CDROM has no label by default, thus its kinda hard to make partitions. And third, even if we do flush a UFS filesystem image off to the cdrom, the geometries will be all wrong, unless your staging disk just happens to have the same
geometries as the CDROM.

My solution is probally not the best. What would be required is an application that simply converts the UFS geometries from the staging disk uses to those used by the cdrom. I didnt really have time to create this so I did it as follows. Im sure I will get lots of flak for this
solution, but it does work.

1) Use dd to grab the first cylinder off the solaris boot cdrom. This contains a valid disk label and VTOC for the cdrom. Once this is created, our limitation is that we must work within the defines of this VTOC. You should be able to use prtvtoc on the cdrom to get a look at this VTOC, but this dosnt work if volmgr is running.

dd if=/dev/dsk/c0t6d0s0 of=cdrom.vtoc bs=512 count=1

2) Now use dd to grab the UFS slices from 1 to 5

for slice in 1 2 3 4 5
do
dd if=/dev/dsk/c0t6d0${slice} of=cdrom.s${slice}
done

3) Create a staging area and copy the parts of the usr filesystem (slice 0) off the cdrom into it. I started by copying cdrom:/export to it and then trimmed out the parts I didnt need like X and openwindows.

4) Add in all the things you need for your disc. For me, this ment a shell script that automated the build process, and images of all the data I wanted to move out. Make sure you donot go beyond te size of the slice 0 on the cdrom you started with. prtvtoc will should you this, if you cant get prtvtoc to work on the cdrom (sometimes it does, sometimes it dosnt) then use

> dd if=/dev/dsk/c0t6d0s0 of=/dev/null bs=512 <

Recordthe exact size of the partition in blocks as you will need it later.

5) Patch the slice 1 image (cdrom.s1) to start your custom application rather then the suninstall. This can be done by finding the break point you wish to use in the file cdrom:/sbin/sysconfig, selecting a unique set of chars in this file ( I used the string #***** S30sysid.net ) and then searching via a hexeditor or emacs in bin mode for that string.

Then find a comment line, change the first # and chars after to point to your script, then add a # line after. Example, I patched my image so the line

#***** S30sysid.net

became

exec build #ysid.net

I then put a script in my staging area into the /usr/bin dir. (the staging area will be made into slice 0 which is hsfs)

6) Using mkisofs or the tools that came with your cdrom burning package (HyCD worked well, as well as Gear) turn your staging area into a hsfs filesystem image, making sure that symbloic links are unmodified. HyCD required changing a default option that would have modified all the links.

7) Using dd, throw away the first block of this image.

dd if=image of=image.data bs=512 skip=1

8) Subtract the block count of slice 0 from the solaris cdrom from the image size above, add one to the answer. Say dd reported for the above step that your image was 500000 blocks, and your solaris cdrom has a size of 787840 for slice 0

787840 - ( 500000 + 1 ) = 287839

9) Feed this number into dd reading from /dev/zero to build a pad file.

dd if=/dev/zero of=image.pad bs=512 count=287839

10) Cat all of the image files together with the VTOC and the UFS slices.

cat cdrom.vtoc > image
cat image.data >> image
cat image.pad >> image
cat cdrom.s1 >> image
..
cat cdrom.s5 >> image

11) Burn this image to the cdrom drive using cdrecord, HyCD, Gear, etc

12) put it in a machine and test boot it and make sure it does what you need.

Thats about it. We have to go through the gyrations because UFS is geometry sensitive. We cannot take a image of a hard disk built UFS as all the cylinder groups will be off. The RIGHT way of doing this would be to build a tool that did the conversion for you and built an image up for burning. But that requires a bit more work, this gets the job done with a
minimal amount of strain. I am working on the above mentioned tool, but so far my progress has been to create lots of coasters. Sun has a tool called MakeDisc that does this job, or something similar, but I do not have a copy of it, so had to develop a method, while under a big gun, to do it in a very compressed amount of time. The only real limitation this
has is you can only store about 400 megs of information, of which around 70 or so are needed by usr in the hsfs partition. Plus you do not have to go through the pain of figuring out which parts of the system need to be moved to the memfs filesystem (cdrom is readonly, so dev, devices, etc need to be linked to /tmp)

If you wish to flame the procedure and tell me that I did it really stupid, please correct me! I would willing stop development of my tools to do same and do it the right way :) But my original question on how to do this went unanswered.

Have Fun
James

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Humor

It is jokingly said that dd stands for "destroy disk" or "delete data", since, being used for low-level operations on hard disks, a small mistake, such as reversing the if and of parameters, may accidentally render the entire disk unusable:

Tips For Linux - How and when to use the dd command

Also Murphy's Law was formulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, and it is the only copy in the universe the effect of a bad spot on the magnetic media is devastating. dd can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: February 10, 2021