~ UNIX and Linux System Administration

Quickly creating large files

I’m surprised how many people never think to do this…. but it makes it quite easy.

If you need a large text file, perhaps with 1,000s of lines (or even bigger) – just use doubling to your advantage! For example, create 10 lines. Then use vi (or other editor) to copy the entire file to itself – now 20 lines. If you remember how a geometric progression goes, you’ll have your 1,000s of lines rather fast:

10 lines…

20 lines…

40 lines…

80 lines…

160 lines…

320 lines…

640 lines…

1280 lines…

2560 lines…

10240 lines…

Ten steps and we’re at 10,000+ lines. In the right editor (vi, emacs, etc.) this could be a macro for even faster doubling. This doubling could also be used at the command line:

cat file.txt >> file.txt

Combined with shell history, that should double nicely – though using an editor would be more efficient (fewer disk reads and writes).

When writing code, often programmers will want to set things off with a line of asterisks, hash marks, dashes, or equals signs. Since I use vi, I like to type in five characters, then copy those five into 10, then copy those 10 and place the result three times. There you have 40 characters just like that.

If only a certain number of characters is needed, use dd:

dd if=/dev/random of=myfile.dat bs=1024 count=10

With this command (and bs=1024), the count is in kilobytes. Thus, the example will create a 10K file. Using the Korn shell, one can use this command to get megabytes:

dd if=/dev/random of=myfile.dat bs=$(( 1024 * 1024 )) count=100

This command will create a 100M file (since bs=1048576 and count=100).

If you want files filled with nulls, just substitute /dev/null for /dev/random in the previous commands.

You could use a word dictionary for words, or a Markov chain for pseudo-prose. In any case, if you only want a certain size, do this:

As you mentioned, /dev/urandom is nonblocking. The article you pointed to was the article on /dev/random. The article makes for very interesting reading.

Under Linux, /dev/random is designed to be free of cryptographic export controls (by not using ciphers to generate randomness), and may block at times to receive enough entropy from the system.

In contrast, /dev/urandom has a feedback loop where it feeds generated entropy back into itself, and it will not block.

Under FreeBSD, two things are notable: 1) the Linux-style randomness generator was replaced by something called the Yarrow algorithm; and 2) /dev/urandom is linked to /dev/random – put another way, neither /dev/random nor /dev/urandom will block.

Both /dev/random and /dev/urandom are available all over: including Solaris, MacOS X, FreeBSD, HP-UX, Tru64, and AIX to name a few.

In summary, if you require the best random number generator possible – use /dev/random. If you require nonblocking I/O or faster number generation, use /dev/urandom. Thirdly, if you’re using FreeBSD it doesn’t matter which you use.

The article is very interesting: if you are interested in mathematics, you simply must read it.

I figured it out… And then later noticed the discussion in the feedback. You have to use URANDOM instead of RANDOM. As far as I can tell, however, RANDOM simply never finishes. I let it run for a long time on a 10 kilobyte file and it never finished.

That’s really interesting! I’ve been trying to write a script to run a disk wipe process (using Jom Garlick’s scrub utility – on all free space), and I tried to use up most of the freespace first to speed up testing by copying a 2.5GB about 80 times.
A colleague suggested there was a way of creating files without filling them, so I Googled away and found this page. Oddly enough, on my Solaris 10 system, the -n parameter actually slows it down! (from 8-18″ to 20-30″ per GB) …
# time mkfile 1g -n -v 1GBspacehog.00
real 0m39.789s
user 0m0.087s
sys 0m4.120s
# time mkfile 1g -v 1GBspacehog.000
real 0m19.876s
user 0m0.058s
sys 0m2.691s

Mei Douthitt

Mei is an experienced UNIX and Linux system administrator, a former Linux distribution maintainer, and author of two books ("Advanced Topics in System Administration" and "GNU Screen: A Comprehensive Manual").