FYI, if this is a windows partition you are trying to clone, use ntfsclone instead of dd. It is smart enough to skip the trash in the unused/free sectors, and can compress the image.
–
psusiMar 15 '12 at 19:04

@psusi: could you tell more in an answer? Yes, this is a w -partitition (actually the whole harddrive is from fresh w machine). I tried this ntfsclone --save-image --output - /dev/sdb3 | bzip2 | ssh m@m.com 'cat > 15032012_w7_3.img.bz2' but getting no content?!
–
user2362Mar 15 '12 at 19:31

Bah, jofel beat me to it. Still leaving though as the answer is slightly different. Also invoking pv in the method I provided lets pv give you progress of how much is left (including time estimates), instead of just how much has transferred so far.
–
PatrickMar 15 '12 at 19:32

why ibs=1M and obs=1M?
–
user2362Mar 16 '12 at 1:26

Larger block sizes (over the default for dd) improve IO to/from physical volumes. While the the value jofel gave (64k) is probably fine, memory is cheap and I usually just set it at 1m.
–
PatrickMar 16 '12 at 8:29

@Patrick I've done some measurements. See my comment on my answer. I measure with obs=1M 1.8 GB/s, with obs=16k 2.8 GB/s from /dev/zero to /dev/null over a pipe. But in our case the queue is usually not the limiting factor, so the discussion is just theoretical...
–
jofelMar 16 '12 at 9:31

@jofel piping to /dev/null is not a valid test. Its size of the blocks to/from the physical volume that matters.
–
PatrickMar 16 '12 at 9:32

bs gives the block size for the transfer. AFAIK, the pipe buffer is 64k big. This is the reason for bs=64k. It maybe better to use e.g. ibs=1M as in Patrix answer for fast disk reading and obs=64k for good pipe performance. See (here)[unix.stackexchange.com/a/11954/15241] for more information. You can install programs in Knoppix based LiveCD systems if you have Internet access. Simply: apt-get update and then apt-get install pv.
–
jofelMar 16 '12 at 8:12

You mean that pipes will fail if you send 50GB with it? How can I see that it fails?
–
user2362Mar 16 '12 at 8:15

1

No, they do not fail. A pipe is limited queue between programs. On Linux, it has 64k. If the queue is full, the writing program on the one side is automatically blocked until the program on the other side reads data from the pipe. Therefore it make sense to write in blocks which fits into the queue. I've done now some pipe speed measurements. I see no big difference between values between obs=64k and obs=8k. Without any proof, it seems that obs=16k works best for me. But in our case, network or disk speed are the limiting factor, so obs= is not really important if it is not too small.
–
jofelMar 16 '12 at 9:23

..yes but what about if I do not specify the setting?
–
user2362Mar 16 '12 at 9:36