Problem

Clojure’s hashing strategy for numbers, sequences/vectors, sets, and maps mimics Java’s. In Clojure, however, it is far more common than in Java to use longs, vectors, sets, maps and compound objects comprised of those components (e.g., a map from vectors of longs to sets) as keys in other hash maps. It appears that Java’s hash strategy is not well-tuned for this kind of usage. Clojure’s hashing for longs, vectors, sets, and maps each suffer from some weaknesses that can multiply together to create a crippling number of collisions.

For example, Paul Butcher wrote a simple Clojure program that produces a set of solutions to a chess problem. Each solution in the set was itself a set of vectors of the form [piece-keyword [row-int col-int]]. Clojure 1.5.1's current hash function hashed about 20 million different solutions to about 20 thousand different hash values, for an average of about 1000 solutions per unique hash value. This causes PersistentHashSet and PersistentHashMap to use long linear searches for testing set/map membership or adding new elements/keys. There is nothing intentionally pathological about these values – they simply happened to expose this behavior in a dramatic way. Others have come across similarly bad performance without any obvious reason why, but some of those cases are likely due to this same root cause.

Change the hash of integers that fit within a long to the return value of longHashMunge (see Longs section of doc for more details)

Change the current multiplier of 31 used for vectors, sequences, and queues to a different constant such as -1640531535 or 524287 (see Vectors section). Also applies to Cons, LazySeq.

For sets, instead of adding together the hash value of the elements, add together the return value of a function xorShift32 called on the hash value of each element (see Sets section)

For maps and records, instead of adding together hash(key) ^ hash(val) for each hash,val pair, instead add together hash(key)^xorShift32(hash(val)) (see Maps section)

No change for any other types, e.g. strings, keywords, symbols, floats, doubles, BigInt's outside of long range, BigDecimal, etc.

Below is a link to a modified version of Paul Butcher's N-queens solver, with extra code for printing stats with several different hash functions. The README has instructions for retrieving and installing locally a version of Clojure modified with one of Mark's proposed alternate hash functions. After that is a link to a patch that implements the proposal above.

Here is a summary of results for some program elapsed times and how spread out the hash values are. Below the table are a few details on how these measurements were made. All times are elapsed times. "meas" means raw measurements of elapsed time, listed in ascending order so you can easily see min and max. "avg" is the average of multiple elapsed times.

Only the N-queens measurements used Leiningen. The compilation measurements used the ant commands shown. The rest were measured using the expressions shown after starting a JVM with the command line:

java -cp clojure.jar clojure.main

with the version of clojure.jar given in the column heading. 5 measurements were taken for each. The raw measurements are given, and the average. Each individual run is intended to be long enough to avoid any concerns about misleading measurements from microbenchmarks.

Open questions

Possible small changes to the proposal for consideration:

Add in some magic constant to the hash of all sets and maps (a different constant for sets than for maps), so that the empty set and empty map have non-0 hash values, and thus nested data structures like #{{}} and {#{} #{}} will hash to non-0 values that are different from each other.

Consider doing something besides the current hash(numerator) ^ hash(denominator) for Ratio values. Perhaps hash(numerator) + xorShift32(hash(denominator)) ? That would avoid the hash of 3/7 and 7/3 being the same, and also avoid the current behavior that hash(3/7) "undoes" the longHashMunge proposed for both the numerator and denominator long values.

Tradeoffs and alternatives

These are discussed throughout Mark's document. A few of these are called out below

Nearly all of the proposals involve additional operations on ints or longs. This is expected to require little additional elapsed time in most hash computations, given that the most significant cost in hashing collections is usually traversing the parts of the data structure, especially if that involves cache misses. Measurements are given above for one proposed set of hash function changes.

Murmur3 is widely used, but does not lend itself well to incremental updates of hash calculations of collections.

A follow-up writeup

I (Alex Miller) have spent a bunch of time looking at this. It probably bears some discussion before updating this page so I will simply append this new section for now.

For me, it was a key thing to restate the problem as: "Slightly different collections frequently produce identical hash codes". The primary use where this is an issue is when using sets that contain collections of similar values, especially numbers.

A summary of my conclusions:

DO NOT change long hashing. Not worth the effort or differences from Java.

DO change the vector hashcode algorithm either by changing the multiplier constant, rehashing the items, or both. (Currently I lean towards only rehashing the items.)

DO change the set hashcode algorithm to rehash the items before summing.

DO change the map hashcode algorithm to to remove the symmetric inclusion of keys and values. My preferred change would be to treat K/V as a 2-element vector and re-use whatever algorithm we use for that.

DO NOT use xorShift32 for rehash - the reliance on xor can easily cause collisions when summing hashcodes in sets.

DO use Murmur3 as rehash algorithm - it's fast, extremely well-tested, commonly used, has great avalanche and bit dispersion properties. Note, this is not for hashing the entire collection, but for rehashing individual element hash codes.

DO NOT change the hashCode() of collection functions; change only the hasheq() algorithms.

CONSIDER rehashing keys in PersistentHashMap.hash() for storage and retrieval - this would yield better hash keys for all users of persistent hash maps and sets, not just for numbers, but also for keywords, strings, symbols, and whatever else you put in them. This could yield shallower trees and better average performance on hash lookup for large trees. Proving this would likely require instrumentation inside PHM and/or developing some more intensive performance studies.

Tracking updates to other libraries for Clojure 1.6 hash changes

Clojure 1.6.0-beta1 is out as of this writing, and the hash method has been decided upon. Now comes updating Clojure contrib libraries and 3rd party Clojure libraries that implement their own customized collections so that their hash function values for custom sets and maps (at least those intended to be clojure.core/= to built-in Clojure sets and maps) have consistent hash values when using Clojure 1.6.

The table below gives some places that could use updating, and status of changes.

Project/Library

What needs updating?

Status

Tests in Clojure itself

Some tests were disabled when the hash changes were first made, with this commit.

Most of them were updated to be independent of Clojure's hash function with this commit, but the following line at or near line 311 in file test/clojure/test_clojure/java_interop.clj is still commented out:

Fixed with this commit. Note that this data structure is a special case, in that it can only contain Long values, thus in Clojure 1.5.1 an earlier it was guaranteed that (.hashCode immutable-bitset) and (hash immutable-bitset) was true for all immutable-bitsets. This is not true in general for other collections, because (.hashCode x) and (hash x) are in general different for Integers, Shorts, Bytes, etc.