Archive

Just realized that the poor performance I was seeing when downloading a file from Rackspace Cloudfiles to a local drive was due to shutil.copyfileobj without specifying a buffer size. As I found out, the default buffer is 16k (just in case you’re running on a 640k server), increasing it to 1MB sped up the process from 17.8 seconds to 0.8 seconds, with 5MB down to 0.3 seconds (for files averaging about 3-4MB).

So remember: always supply a buffer size, and the 60x speed increase can be yours too!

A project I’ve worked on over the last few years is now online at http://mdid.cit.jmu.edu/snp/. It is a collection of oral history recordings and transcripts, delivered through a custom interface built on MDID3.