Categories

Recent Tweets

I developed at Flexiant a utility called sparsecopy, which we use to copy around sparse files (and indeed to make non-sparse files sparse, and to copy sparse files to block devices). In essence, it’s a version of dd which supports sparse files better. It’s hidden deep within Flexiant’s web site, and as we released it under the GPLv2, I thought I’d put it up here too. That link will take you to a tarball, which which you can either do the normal make, make install on, or build a Debian / Ubuntu package using debuild. It’s pretty self-explanatory, but here are the options:

Usage

sparsecopy [OPTIONS] SOURCE DEST

Options

-o, --overlay

Overlay existing file DEST (rather than truncating)

-n, --nocheck

Skip blank check

-q, --quiet

Quiet

-m, --max SIZE

Copy SIZE bytes maximum from SOURCE

-b, --blocksize SIZE

Use SIZE blocksize in bytes (default 512)

-s, --seek POS

Start writing SOURCE at position POS in DEST

-f, --finalsize SIZE

Set size of DEST to SIZE when done (no effect on block devices)

-c, --check

Check the first block in the destination is zero, else abort

-p, --progress

Display progress indicator on stderr

-h, --help

Display usage

Notes

Sparsecopy copies from a source file (either sparse or non-sparse) to a destination file making the destination file sparse on the way. If the destination file already exists then the source file can be superimposed on top of it by using the -o switch (ignored if the destination file doesn’t exist); in this case sparsecopy will (unless -n is specified as well) check that any blocks to be overlayed with zero (i.e. made sparse) are already zero in the destination images. When used with block devices, sparsecopy will not affect the size of the device.

SOURCE can be "-" to indicate stdin. SOURCE or DEST can be raw devices if you wish. If you want to make a sparse file, either use /dev/zero for SOURCE and -m, or (quicker) use /dev/null for SOURCE and -f.

SIZE and POS can be specified in blocks (default), or use the following suffixes:

Note that blocksize=1024 will set blocksize to 1024 512 byte blocks (use 1024B if this is not what you mean). Also note that disk capacity is often measured using decimal megabytes etc.; we do not adopt this convention for compatibility with dd.

Hi, I have been dealing with very large sparse files. Your script is very helpful.

I have found that the changes in the latest versions of cp 8.13 onwards (using FIEMAP), really helps because it only reads the actual allocated blocks. Very helpful when you are dealing with 1TB+ sparse files!

I haven’t done so to date. I am aware of the changes to cp, and it would be relatively easily to change sparsecopy to use a similar strategy, which would speed up reading files that are already sparse.