Thank you Jake, also Aaron & Peter for your help :-)
It was the 1801 bug, solved in RC2 SVN Snapshot!
Max
Jake Luciani wrote:
> Max this was a bug fixed recently in 0.7 branch
>
> https://issues.apache.org/jira/browse/CASSANDRA-1801
>
> fixed now in RC2
>
> -Jake
>
> On Tue, Dec 7, 2010 at 8:11 AM, Max wrote:
>
>> As far as i can see, Lucandra already uses batch_mutations.
>>
>> https://github.com/tjake/Lucandra/blob/master/src/lucandra/IndexWriter.java#L263
>>
>> https://github.com/tjake/Lucandra/blob/master/src/lucandra/CassandraUtils.java#L371
>>
>> IndexWriter.addDocument() merges all fields to a mutioation map.
>> In addition instead of "autoCommit" (commit each doc), i commit only every
>> 10 documents. Where can i monitor incoming requests to cassandra?
>> WriteCount and MutationCount (monitored by jconsole) didn't change
>> obviously.
>>
>> I had problems to open the jrockit heapdump with MAT, but found "jrockit
>> mission control" instead. Unfortunately i'm not confident using it.
>>
>> Here my observations:
>> While heapByteBuffer was growing (~200mb) and flushed during client insert
>> the byte[] was growing permanetly.
>> http://oi51.tinypic.com/2uhbdp3.jpg
>>
>> I used TypeGraph to analyze the byte[] but i'm not sure how to interpret:
>> http://oi53.tinypic.com/y2d1i.jpg
>>
>> Thank you!
>> Max
>>
>>
>> Aaron Morton wrote:
>>
>>> Jake or anyone else got experience bulk loading into Lucandra ?
>>>
>>> Or does anyone have experience with JRocket ?
>>>
>>> Max, are you sending one document at a time into lucene. Can you send
>>> them in batches (like solr), if so does it reduce the
>>> amount of requests going to cassandra?
>>>
>>> Also, cassandra.bat is configured with XX:+HeapDumpOnOutOfMemoryError so
>>> you should be able to take a look at where all the memory if
>>> going. Riptano
>>> blog points to http://www.eclipse.org/mat/ also see
>>> http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gdyrr
>>>
>>> Hope that helps.
>>>
>>> Aaron
>>>
>>> On 07 Dec, 2010,at 09:17 AM, Aaron Morton
>>> wrote:
>>>
>>> Accidentally sent to me.
>>>
>>> Begin forwarded message:
>>> From: Max
>>> Date: 07 December 2010 6:00:36 AM
>>> To: Aaron Morton
>>> Subject: Re: Re: Re: Cassandra 0.7 beta 3 outOfMemory (OOM)
>>>
>>> Thank you both for your answer!
>>> After several tests with different parameters we came to the conclusion
>>> that it must be a bug.
>>> It looks very similar to:
>>> https://issues.apache.org/jira/browse/CASSANDRA-1014
>>>
>>> For both CFs we reduced thresholds:
>>> - memtable_flush_after_mins = 60 (both CFs are used permanently,
>>> therefore other thresholds should trigger first)
>>> - memtable_throughput_in_mb = 40
>>> - memtable_operations_in_millions = 0.3
>>> - keys_cached = 0
>>> - rows_cached = 0
>>>
>>> - in_memory_compaction_limit_in_mb = 64
>>>
>>> First we disabled caching, later we disabled compacting and after that we
>>> set
>>> commitlog_sync: batch
>>> commitlog_sync_batch_window_in_ms: 1
>>>
>>> But our problem still appears:
>>> During inserting files with Lucandra memory usage is slowly growing
>>> until OOM crash after about 50 min.
>>> @Peter: In our latest test we stopped writing suddenly but cassandra
>>> didn\'t relax and remains even after minutes on ~90% heap usage.
>>> http://oi54.tinypic.com/2dueeix.jpg
>>>
>>> With our heap calculation we should need:
>>> 64 MB * 2 * 3 + 1 GB = 1,4 GB
>>> All recent tests we run with 3 GB. I think that should be ok for a test
>>> machine.
>>> Also consistency level is one.
>>>
>>> But Aaron is right, Lucandra produces even more than 200 inserts/s.
>>> My 200 documents per second are about 200 operations (writecount) on
>>> first CF and about 3000 on second CF.
>>>
>>> But even with about 120 documents/s cassandra crashes.
>>>
>>>
>>> Disk I/O monitored with Windows performance admin tools is on both
>>> discs moderate (commitlog is on seperate harddisc).
>>>
>>>
>>> Any ideas?
>>> If it's really a bug, in my opinion it's very critical.
>>>
>>>
>>>
>>> Aaron Morton wrote:
>>>
>>> I remember you have 2 CF's but what are the settings for:
>>>>
>>>> - memtable_flush_after_mins
>>>> - memtable_throughput_in_mb
>>>> - memtable_operations_in_millions
>>>> - keys_cached
>>>> - rows_cached
>>>>
>>>> - in_memory_compaction_limit_in_mb
>>>>
>>>> Can you do the JVM Heap Calculation here and see what it says
>>>> http://wiki.apache.org/cassandra/MemtableThresholds
>>>>
>>>> What Consistency Level are you writing at? (Checking it's not Zero)
>>>>
>>>> When you talk about 200 inserts per second is that storing 200 documents
>>>> through lucandra or 200 request to cassandra. If it's the first option I
>>>> would assume that would generate a lot more actual requests into
>>>> cassandra.
>>>> Open up jconsole and take a look at the WriteCount settings for the CF's
>>>> http://wikiapache.org/cassandra/MemtableThresholds
>>>>
>>>>
>>>> You could also try setting the compaction thresholds to 0 to disable
>>>> compaction while you are pushing this data in. Then use node tool to
>>>> compact and turn the settings back to normal. See cassandra.yam for
>>>> more info.
>>>>
>>>> I would have thought you could get the writes through with the setup
>>>> you've described so far (even though a single 32bit node is unusual).
>>>> The best advice is to turn all the settings down (e.g. caches off,
>>>> mtable flush 64MB, compaction disabled) and if it still fails try:
>>>>
>>>> - checking your IO stats, not sure on windows but JConsole has some IO
>>>> stats. If your IO cannot keep up then your server is not fast enough
>>>> for your client load.
>>>> - reducing the client load
>>>>
>>>> Hope that helps.
>>>> Aaron
>>>>
>>>>
>>>> On 04 Dec, 2010,at 05:23 AM, Max wrote:
>>>>
>>>> Hi,
>>>>
>>>> we increased heap space to 3 GB (with JRocket VM under 32-bit Win with
>>>> 4 GB RAM)
>>>> but under "heavy" inserts Cassandra is still crashing with OutOfMemory
>>>> error after a GC storm.
>>>>
>>>> It sounds very similar to
>>>> https://issues.apache.org/jira/browse/CASSANDRA-1177
>>>>
>>>> In our insert-tests the average heap usage is slowly growing up to the
>>>> 3 GB border (jconsole monitor over 50 min
>>>> http://oi51.tinypic.com/k12gzd.jpg) and the CompactionManger queue is
>>>> also constantly growing up to about 50 jobs pending
>>>>
>>>> We tried to decrease CF memtable threshold but after about half a
>>>> million inserts it's over.
>>>>
>>>> - Cassandra 0.7.0 beta 3
>>>> - Single Node
>>>> - about 200 inserts/s ~500byte - 1 kb
>>>>
>>>>
>>>> Is there no other possibility instead of slowing down inserts/s ?
>>>>
>>>> What could be an indicator to see if a node works stable with this
>>>> amount of inserts?
>>>>
>>>> Thank you for your answer,
>>>> Max
>>>>
>>>>
>>>> Aaron Morton :
>>>>
>>>> Sounds like you need to increase the Heap size and/or reduce the
>>>>> memtable_throughput_in_mb and/or turn off the internal caches. Normally
>>>>> the binary memtable thresholds only apply to bulk load
>>>>> operations and it's
>>>>> the per CF memtable_* settings you want to change. I'm not familiar with
>>>>> lucandra though.
>>>>>
>>>>> See the section on JVM Heap Size here
>>>>> http://wiki.apache.org/cassandra/MemtableThresholds
>>>>>
>>>>> Bottom line is you will need more JVM heap memory.
>>>>>
>>>>> Hope that helps.
>>>>> Aaron
>>>>>
>>>>> On 29 Nov, 2010,at 10:28 PM, cassandra@ajowa.de wrote:
>>>>>
>>>>> Hi community,
>>>>>
>>>>> during my tests i had several OOM crashes.
>>>>> Getting some hints to find out the problem would be nice.
>>>>>
>>>>> First cassandra crashes after about 45 min insert test script.
>>>>> During the following tests time to OOM was shorter until it started to
>>>>> crash
>>>>> even in "idle" mode.
>>>>>
>>>>> Here the facts:
>>>>> - cassandra 0.7 beta 3
>>>>> - using lucandra to index about 3 million files ~1kb data
>>>>> - inserting with one client to one cassandra node with about 200 files/s
>>>>> - cassandra data files for this keyspace grow up to about 20 GB
>>>>> - the keyspace only contains the two lucandra specific CFs
>>>>>
>>>>> Cluster:
>>>>> - cassandra single node on windows 32bit, Xeon 2,5 Ghz, 4GB RAM
>>>>> - java jre 1.6.0_22
>>>>> - heap space first 1GB, later increased to 1,3 GB
>>>>>
>>>>> Cassandra.yaml:
>>>>> default + reduced "binary_memtable_throughput_in_mb" to 128
>>>>>
>>>>> CFs:
>>>>> default + reduced
>>>>> min_compaction_threshold: 4
>>>>> max_compaction_threshold: 8
>>>>>
>>>>>
>>>>> I think the problem appears always during compaction,
>>>>> and perhaps it is a result of large rows (some about 170mb).
>>>>>
>>>>> Are there more options we could use to work with few memory?
>>>>>
>>>>> Is it a problem of compaction?
>>>>> And how to avoid?
>>>>> Slower inserts? More memory?
>>>>> Even fewer memtable_throuput or in_memory_compaction_limit?
>>>>> Continuous manual major comapction?
>>>>>
>>>>> I've read
>>>>>
>>>>> http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors
>>>>> - row_size should be fixed since 0.7 and 200mb is still far away from
>>>>> 2gb
>>>>> - only key cache is used a little bit 3600/20000
>>>>> - after a lot of writes cassandra crashes even in idle mode
>>>>> - memtablesize was reduced and there are only 2 CFs
>>>>>
>>>>> Several heapdumps in MAT show 60-99% heapusage of compaction thread.
>>>>>
>>>>
>