Turtle: Identifying frequent k-mers with cache-efficient algorithms

R.S. Roy, D. Bhattacharya and A. Schliep

Bioinformatics 2014, 14:30, 1950–7.

Counting the frequencies of k-mers in read libraries is often a
first step in the analysis of high-throughput sequencing
experiments. Infrequent k-mers are assumed to be a result of sequencing
errors. The frequent k-mers constitute a reduced but error-free
representation of the experiment, which can inform read error
correction or serve as the input to de novo assembly methods. Ideally,
the memory requirement for counting should be linear in the number of frequent
k-mers and not in the, typically much larger, total number of
k-mers in the read library.
We present a novel method that balances time, space and accuracy
requirements to efficiently extract frequent k-mers even for high
coverage libraries and large genomes such as human. Our method is
designed to minimize cache-misses in a cache-efficient manner by using
a Pattern-blocked Bloom filter to remove infrequent k-mers from
consideration in combination with a novel sort-and-compact
scheme, instead of a Hash, for the actual counting. While this
increases theoretical complexity, the savings in cache misses reduce
the empirical running times. A variant can resort to a counting Bloom
filter for even larger savings in memory at the expense of false
negatives in addition to the false positives common to
all Bloom filter based approaches. A comparison to the
state-of-the-art shows reduced memory requirements and running times.
Note that we also provide the first competitive method to count k-mers up to size 64.