In singlepass indexing, checking for enough free memory is insufficient

Details

Description

Single-pass tries to use as much memory as possible for mini-inverted indices (flushes). It uses some Java code to guess how much memory is left.
When JVM has allocated all memory, and when only 70% free, flush().

However, Java's memory management isn't reliable. We can easily get out of memory errors, particularly in Hadoop mode, and often for block indexing. Java6 makes this problem worse as it will throw OutOfMemoryError earlier than Java 5 would.