During performance testing being run on our 4 node C=
assandra 1.1.5 cluster, we are seeing warning logs about the heap being alm=
ost full – [1]. I’m trying to figure out why, and how to preven=
t it.

The tests are being run on a Cassandra ring consisti=
ng of 4 dedicated boxes with 32 GB of RAM each.

The heap size is set to 8 GB as recommended.

=
o:p>

All the other relevant settings I know off are the d=
efault ones:

- &=
nbsp;
memtable_total_space_in_mb is not set in the yaml, =
so should default to 1/3rd the heap size.

- &=
nbsp;
They key cache should be 100 MB at the most. I chec=
ked the key cache the day after the tests were run via nodetool info, and i=
t reported 4.5 MB being used.

- &=
nbsp;
row cache is not being used

- &=
nbsp;
I summed up the bloom filter usage reported by node=
tool cfstats in all the CFs and it was under 50 MB.

The resident size of t=
he cassandra process accd to top is 8.4g even now. Did a heap histogram usi=
ng jmap, but not sure how to interpret those results usefully – [2].<=
o:p>

Performance test detai=
ls:

-&nb=
sp;
The test is wr=
ite only, and is writing relatively large amount of data to one CF.

-&nb=
sp;
There is some =
other traffic that is constantly on that writes smaller amounts of data to =
many CFs, and does some reads.

The total number of CF=
s are 114, but quite a few of them are not used.

Thanks,

Arindam

[1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1:&nbsp=
; WARN GCInspector.java (line 145) Heap is 0.8287082580489245 full. Y=
ou may need to reduce memtable and/or cache sizes. Cassandra will now=
flush up to the two largest memtables to free up
memory. Adjust flush_largest_memtables_at threshold in cassandra.yam=
l if you don't want Cassandra to do this automatically