From user-return-33638-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Apr 22 07:43:16 2013
Return-Path:
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
by minotaur.apache.org (Postfix) with SMTP id 63864F236
for ; Mon, 22 Apr 2013 07:43:16 +0000 (UTC)
Received: (qmail 44221 invoked by uid 500); 22 Apr 2013 07:43:14 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 43940 invoked by uid 500); 22 Apr 2013 07:43:13 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 43918 invoked by uid 99); 22 Apr 2013 07:43:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Apr 2013 07:43:12 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of arodrime@gmail.com designates 209.85.215.49 as permitted sender)
Received: from [209.85.215.49] (HELO mail-la0-f49.google.com) (209.85.215.49)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Apr 2013 07:43:08 +0000
Received: by mail-la0-f49.google.com with SMTP id ei20so1259425lab.36
for ; Mon, 22 Apr 2013 00:42:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=x-received:mime-version:in-reply-to:references:from:date:message-id
:subject:to:content-type;
bh=KtOONI6HO6qjOxb5YqC6ca4hWW45+hAdlxwbz1FI7GI=;
b=Xo1DU6ZaI+vBEwpU4msY+j3QW3r28YNz8xNhmg+kdnnGT7Ooor7RIXnl/y5o48seD+
CyE5okkydxybDiu7oQ0L9iIFyY0544DCOnodU1yG2kEHn5ul7YBQj/rtAROl81tq/1FJ
Gpr6KVQpPV/kk7IJm/pvj8oBLjzCM3+d2bHDwS07VRZtWMZmvkI9NeC0uZy6Xv000+fY
JfodTvlEipe24sD3Jzq6Z+dXYaQa16F1wcR2dB9Yi8396zEX/URAmgNu2FbEX4l7wgTV
ZaJ7kEweYxmXqy7X0OjUTimtq2wMSt/J7L8M3Frw8UGf0nnHb8G43zwqQjTqB/YRZJa2
kJEQ==
X-Received: by 10.152.28.130 with SMTP id b2mr12666831lah.51.1366616566947;
Mon, 22 Apr 2013 00:42:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.112.168.1 with HTTP; Mon, 22 Apr 2013 00:42:26 -0700 (PDT)
In-Reply-To:
References: <3370D3BC-48D6-4112-B800-92BADE80946D@yahoo.com>
From: Alain RODRIGUEZ
Date: Mon, 22 Apr 2013 09:42:26 +0200
Message-ID:
Subject: Re: Advice on memory warning
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=089e0160b6bc04693604daee3794
X-Virus-Checked: Checked by ClamAV on apache.org
--089e0160b6bc04693604daee3794
Content-Type: text/plain; charset=ISO-8859-1
I would have said the exact opposite, but I am not really sure.
I have configured the threshold to 80% of the heap since I have 8GB heap. I
think the purpose of this threshold is to keep a security margin to avoid
OOMing. C* can be configured with 1GB heap so the margin is about 250MB. On
a 8GB heap you have 2GB unused heap (since memtables will be forced to
flush when you reach 6GB). My guess is that the more memory you have in the
heap, the more you can tune this threshold high.
Yet, this isn't a solution to anything since you shouldn't reach such a
high heap used, but more a workaround.
Once again this is my interpretation, I can be wrong.
Alain
2013/4/21 Derek Williams
> The CMS compaction threshold is usually set to 75% as well, it might help
> to set it lower to 70% to see if that resolves these warnings as Cassandra
> will start CMS GC before it hits the 75% warning.
>
> There is also a setting to lower the max amount of memory used for
> compacting each row. This may cause compactions to take longer though as
> each row that surpasses this limit will need to be compacted on disk
> instead of in memory.
>
>
> On Fri, Apr 19, 2013 at 9:00 AM, Michael Theroux wrote:
>
>> Hello,
>>
>> We've recently upgraded from m1.large to m1.xlarge instances on AWS to
>> handle additional load, but to also relieve memory pressure. It appears to
>> have accomplished both, however, we are still getting a warning, 0-3 times
>> a day, on our database nodes:
>>
>> WARN [ScheduledTasks:1] 2013-04-19 14:17:46,532 GCInspector.java (line
>> 145) Heap is 0.7529240824406468 full. You may need to reduce memtable
>> and/or cache sizes. Cassandra will now flush up to the two largest
>> memtables to free up memory. Adjust flush_largest_memtables_at threshold
>> in cassandra.yaml if you don't want Cassandra to do this automatically
>>
>> This is happening much less frequently than before the upgrade, but after
>> essentially doubling the amount of available memory, I'm curious on what I
>> can do to determine what is happening during this time.
>>
>> I am collecting all the JMX statistics. Memtable space is elevated but
>> not extraordinarily high. No GC messages are being output to the log.
>>
>> These warnings do seem to be occurring doing compactions of column
>> families using LCS with wide rows, but I'm not sure there is a direct
>> correlation.
>>
>> We are running Cassandra 1.1.9, with a maximum heap of 8G.
>>
>> Any advice?
>> Thanks,
>> -Mike
>
>
>
>
> --
> Derek Williams
>
--089e0160b6bc04693604daee3794
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I would have said the exact opposite, but I am not really =
sure.

I have configured the threshold to 80% of th=
e heap since I have 8GB heap. I think the purpose of this threshold is to k=
eep a security margin to avoid OOMing. C* can be configured with 1GB heap s=
o the margin is about 250MB. On a 8GB heap you have 2GB unused heap (since =
memtables will be forced to flush when you reach 6GB). My guess is that the=
more memory you have in the heap, the more you can tune this threshold hig=
h.

Yet, this isn't a solution to anything =
since you shouldn't reach such a high heap used, but more a workaround.=

The CMS compaction thr=
eshold is usually set to 75% as well, it might help to set it lower to 70% =
to see if that resolves these warnings as Cassandra will start CMS GC befor=
e it hits the 75% warning.

There is also a setting to lower the max amount o=
f memory used for compacting each row. This may cause compactions to take l=
onger though as each row that surpasses this limit will need to be compacte=
d on disk instead of in memory.

We've recently upgraded from m1.large to m1.xlarge instances on AWS to =
handle additional load, but to also relieve memory pressure. =A0It appears =
to have accomplished both, however, we are still getting a warning, 0-3 tim=
es a day, on our database nodes:

WARN [ScheduledTasks:1] 2013-04-19 14:17:46,532 GCInspector.java (line 145)=
Heap is 0.7529240824406468 full. =A0You may need to reduce memtable and/or=
cache sizes. =A0Cassandra will now flush up to the two largest memtables t=
o free up memory. =A0Adjust flush_largest_memtables_at threshold in cassand=
ra.yaml if you don't want Cassandra to do this automatically

This is happening much less frequently than before the upgrade, but after e=
ssentially doubling the amount of available memory, I'm curious on what=
I can do to determine what is happening during this time.

I am collecting all the JMX statistics. =A0Memtable space is elevated but n=
ot extraordinarily high. =A0No GC messages are being output to the log.

These warnings do seem to be occurring doing compactions of column families=
using LCS with wide rows, but I'm not sure there is a direct correlati=
on.