dd

dd is a command line tool used to copy and convert data between files, devices, separate computers and so forth. dd is an abbreviation for data definition and is said to be reminiscent of IBM's Job Control Language (used on mainframes). dd has many uses and can be combined with other commands like netcat and gz to create one line disk imaging or backup solutions.

Using dd to write data to a hard disk

This example uses /dev/urandom as the input and outputs that data to the hard disk at /dev/hda1. This will write over that disk with random data, effectively wiping it.

dd if=/dev/urandom of=/dev/hda1

Using dd to create an ISO image

In this example we'll use dd to create an ISO image from a CD in the CD drive. We specify the input file to be /dev/cdrom (or wherever our cd drive is) and the output file to be a .iso. bs=2k tells dd to use a block size of 2000 (standard for ISO images).

dd if=/dev/cdrom of=image.iso bs=2k

Using dd to create an image of a hard disk

For backup purposes, you may want to create an image of your hard disk. In this example we'll tell dd to use /dev/sdb1 as the input file and we'll output that to disk.img. To demonstrate, I added a 100MB virtual hard disk to my virtual machine. I added a folder and some files to this new disk so that I can mount the newly created image and we can see that everything worked properly.

dd if=/dev/sdb of=disk.img
[root@blackfedora ~]# mount /dev/sdb /mnt/test [root@blackfedora ~]# ls -R /mnt/test /mnt/test: lost+found stuff test.txt /mnt/test/lost+found: /mnt/test/stuff: file1 file2 file3 [root@blackfedora ~]# umount /mnt/test [root@blackfedora ~]# dd if=/dev/sdb of=disk.img 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 43.2852 s, 2.4 MB/s [root@blackfedora ~]# ls -lh total 101M -rw-r--r-- 1 root root 100M 2009-03-08 15:16 disk.img [root@blackfedora ~]# mount disk.img /mnt/test -o loop [root@blackfedora ~]# ls -R /mnt/test /mnt/test: lost+found stuff test.txt /mnt/test/lost+found: /mnt/test/stuff: file1 file2 file3 [root@blackfedora ~]#

As you can see from the output, /dev/sdb was copied to disk.img. When we mount disk.img, you can see that all the files are intact.

Combining with gzip to compress the image

The last example was perfect for small backup jobs, but what if the hard drive your copying is several gigabytes? Or what if it's mostly empty? Since dd does a simple bit for bit copy, you could backup a 100GB hard drive that only actually contains 1GB of data and the end result would still be 100GB. To compress the image we create, we'll ask gzip for some help and see what happens. We'll use the same 100MB virtual drive from the previous example and see how much we can compress it.

dd if=/dev/sdb | gzip -9 > disk.img.gz
[root@blackfedora ~]# mount /dev/sdb /mnt/test [root@blackfedora ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 8.9G 4.3G 4.6G 49% / tmpfs 505M 88K 505M 1% /dev/shm /dev/sdb 97M 5.6M 87M 7% /mnt/test [root@blackfedora ~]#

As you can see, only 5.6MB of the 100MB disk are being used. Using our new commmand above, we get the following:

[root@blackfedora ~]# dd if=/dev/sdb | gzip -9 > disk.img.gz 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 328.646 s, 319 kB/s [root@blackfedora ~]# ls -lh total 120K -rw-r--r-- 1 root root 115K 2009-03-08 16:01 disk.img.gz [root@blackfedora ~]#

Piping our output to gzip provided a huge benefit in this case. Our new disk image is only 115K. Gzip compressed all the empty space on the hard disk so we're only backing up things that we care about.

Restoring an image taken with dd

Now that we know how to create an image and even compress it to save space we should look at restoring that image back to the hard disk. For this example, I'm going to erase everything on my virtual hard drive and then restore it from the compressed image we created in the last example. This time, we use gunzip -dc to decompress the file and write the contents to stdout for piping.

gunzip -dc disk.img.gz | dd of=/dev/sdb
[root@blackfedora ~]# gunzip -dc disk.img.gz | dd of=/dev/sdb 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 156.083 s, 672 kB/s [root@blackfedora ~]# mount /dev/sdb /mnt/test [root@blackfedora ~]# ls -R /mnt/test /mnt/test: lost+found stuff test.txt /mnt/test/lost+found: /mnt/test/stuff: file1 file2 file3 [root@blackfedora ~]#

It worked. We used gzip to decompress the file and piped that to dd which wrote the contents to /dev/sdb. We remount it and a quick ls -R shows us that all our files have been restored.

A quick introduction to using netcat

We'll take a brief detour to learn a bit about netcat. Netcat is a command that allows you to transfer data from one machine to another. In this example, we'll use netcat to just transfer lines of text from one machine to another.

On the target system:

nc -l -p 7000

On the source system:

nc 12.34.56.78 7000

SourceTarget
[root@blackfedora ~]# nc 12.34.56.78 7000 Testing. 1 2 3. This is how we use netcat to transfer data over the network [root@blackfedora ~]# [root@fedoraten ~]# nc -l 7000 Testing. 1 2 3. This is how we use netcat to transfer data over the network [root@fedoraten ~]#

As you can see, everything we typed on the source machine was transferred over the network, a line at a time, to the target machine.

Using netcat to transfer a file

For this example, we'll take usage of netcat one step farther and attempt to send a file over the network.

On the target system:

nc -l -p 7000 > test.txt

On the source system:

nc 12.34.56.78 7000 < test.txt

SourceTarget
[root@blackfedora ~]# emacs test.txt [root@blackfedora ~]# cat test.txt This is a test file for using netcat. [root@blackfedora ~]# nc 12.34.56.78 7000 < test.txt [root@blackfedora ~]# [root@fedoraten ~]# cat test.txt cat: test.txt: No such file or directory [root@fedoraten ~]# nc -l 7000 > test.txt [root@fedoraten ~]# cat test.txt This is a test file for using netcat. [root@fedoraten ~]#

Again, the file was transferred over the network and we can view its contents on the target machine.

Using netcat to transfer a file with on the fly compression

Going another step further, what if the file you want to transfer is very large and you're worried about how long it's going to take? To solve this, we'll use gzip to compress and decompress the file on the fly so that we end up with the file uncompressed on the target machine.

On the target system:

gzip -c test.txt | nc -l -p 7000

On the source system:

nc 12.34.56.78 7000 | gzip -dc > test.txt

SourceTarget
[root@blackfedora ~]# gzip -c test.txt | nc 12.34.56.78 7000 [root@blackfedora ~]# [root@fedoraten ~]# nc -l 7000 | gzip -dc > test.txt [root@fedoraten ~]# cat test.txt This is a test file for using netcat. [root@fedoraten ~]#

This time, the file was compressed as it flew through the ether to save on bandwidth and transfer time, but ended up uncompressed at the completion of the command. For this simple text file, we probably didn't save any time or bandwidth, but we could have just as easily been transferring a large mp3 or video file.

Combining dd, netcat and gzip to backup a hard drive to a remote location

Now, we'll use our knowledge of netcat to backup our hard drive to a remote location. Backing up to a remote location is always a good idea, and these dd commands will help us a lot. On the target system, we're going to start netcat and just have it send its output to disk.img.gz. That way, we'll have a compressed image of our hard drive stored in a remote location. On the source machine, we'll use dd to copy the drive, gzip it and then pipe it to netcat to send it to the target machine.

On the target system:

nc -l 7000 > disk.img.gz

On the source system:

dd if=/dev/sdb | gzip -9 | nc 12.34.56.78 7000

SourceTarget
[root@blackfedora ~]# dd if=/dev/sdb | gzip -9 | nc 12.34.56.78 7000 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 24.6387 s, 4.3 MB/s [root@blackfedora ~]# [root@fedoraten ~]# nc -l 7000 > disk.img.gz [root@fedoraten ~]# ls -lh disk.img.gz -rw-r--r-- 1 root root 115K 2009-03-08 17:55 disk.img.gz [root@fedoraten ~]#

Now we have a backup of our hard drive stored somewhere other than our local machine in case something ever happens to it.

Using dd, gzip and netcat to copy an image over the network

Now, let's combine all the things we've learned about each command to do something really cool! In this example, we'll use dd in combination with gzip and netcat to copy an image from a hard disk on one machine to a hard disk on another machine. gzip will be used to compress and decompress the image on the fly to keep bandwidth usage down. We start by running the first command below on the target system. This tells netcat to listen on port 7000 and to pipe its output to gzip which will, in turn, decompress that data and send its output to dd for writing to the disk. Once that command is running on the target system, we use the next command on the source system to start the transfer. This command starts copying the drive using dd, piping that output to gzip which is then piped to netcat to be sent to the remote machine.

On the target system:

nc -l 7000 | gzip -dc | dd of=/dev/sdb

On the source system:

dd if=/dev/sdb | gzip -9 | nc 12.34.56.78 7000

SourceTarget
[root@blackfedora ~]# dd if=/dev/sdb | gzip -9 | nc 12.34.56.78 7000 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 362.239 s, 289 kB/s [root@blackfedora ~]# [root@fedoraten ~]# nc -l 7000 | gzip -dc | dd of=/dev/sdb 204800+0 records in 204800+0 records out 104857600 bytes (105 MB) copied, 444.734 s, 236 kB/s [root@fedoraten ~]# !mount mount /dev/sdb /mnt/test [root@fedoraten ~]# ls -R /mnt/test /mnt/test: lost+found stuff test.txt /mnt/test/lost+found: /mnt/test/stuff: file1 file2 file3 [root@fedoraten ~]#

As the output shows, the disk was successfully copied over the network to the remote machine with all the files intact. We've just copied all the data on one hard drive to another hard drive somewhere on the internet, all without even leaving our seats! That is very impressive.



This tutorial created by Justin Bennett in March of 2009