If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try [[ http://fastercsv.rubyforge.org/ | FasterCSV ]] instead of the stock CSV (import 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.

+

If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try [[ http://fastercsv.rubyforge.org/ | FasterCSV ]](require 'faster_csv') instead of the stock CSV (require 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.

Below is the time to load different numbers of row using the two methods on a 2.8Ghz Quad core machine with 3GB RAM. The second method seems quicker. Note that these are just based on single loads and are intended to be a guide rather than a rigorous analysis of the methods!

Rows

1M

2M

3M

Method 1

20s

53s

103s

Method 2

15s

41s

86s

Ruby

Note on CSV Libraries

If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try FasterCSV (require 'faster_csv') instead of the stock CSV (require 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.