From user-return-25254-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Apr 2 15:52:17 2012
Return-Path:
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
by minotaur.apache.org (Postfix) with SMTP id 50C049EAF
for ; Mon, 2 Apr 2012 15:52:17 +0000 (UTC)
Received: (qmail 88840 invoked by uid 500); 2 Apr 2012 15:52:14 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 88755 invoked by uid 500); 2 Apr 2012 15:52:14 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 88747 invoked by uid 99); 2 Apr 2012 15:52:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2012 15:52:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of leimy2k@gmail.com designates 209.85.210.172 as permitted sender)
Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2012 15:52:07 +0000
Received: by iazz13 with SMTP id z13so5269939iaz.31
for ; Mon, 02 Apr 2012 08:51:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=yELlgjpLRvflDvkrdj1TN6t/XcDEGf3+U1tlZ8N8/vg=;
b=dNv8ruzVrZ0vyZ5guTFW16pqrlepWP7CQRgQBAcyGsSUvr6g4XgzcKe5LsOHyBoJY6
MSHfmviXfbgDWRHbb5oCdkJ03wZhZ+SIF09EclroQYtms2oJvahR8uGrGR3fXzkJkOG4
/czzemvtz6LLlwLsffJgyrD5ezsH835oJTJiIlMtl45s9IU7NcXVLbcUZf3e+hg/1gzF
HKWpLy/2CZ9wI23JPT1X5fXV62kLdkQEvv5nXLmxaIFpwl4Y7uHjEU0/r5If0WN9r6tg
ymqtGWPRJLn2HDesqw0k/4fHw9aKRPRvYMHDc+3ZgAOntx3YL/Cc3QioxW2Dog7yM3op
a6mA==
MIME-Version: 1.0
Received: by 10.42.145.72 with SMTP id e8mr4922668icv.0.1333381906717; Mon, 02
Apr 2012 08:51:46 -0700 (PDT)
Received: by 10.42.144.67 with HTTP; Mon, 2 Apr 2012 08:51:46 -0700 (PDT)
In-Reply-To: <6019AFCD-ACBB-4E14-A1DB-73543FA915E3@thelastpickle.com>
References: <1612635136.143108.1333061537915.JavaMail.root@mail.remilon.com>
<6019AFCD-ACBB-4E14-A1DB-73543FA915E3@thelastpickle.com>
Date: Mon, 2 Apr 2012 08:51:46 -0700
Message-ID:
Subject: Re: really bad select performance
From: David Leimbach
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e8ef2e65af504bcb42ab0
--90e6ba6e8ef2e65af504bcb42ab0
Content-Type: text/plain; charset=ISO-8859-1
This is all very hypothetical, but I've been bitten by this before.
Does row_loaded happen to be a binary or boolean value? If so the
secondary index generated by Cassandra will have at most 2 rows, and
they'll be REALLY wide if you have a lot of entries. Since Cassandra
doesn't distribute columns over rows, those potentially very wide index
rows, and their replicas, must live in SSTables in their entirety on the
nodes that own them (and their replicas).
Even though you limit 1, I'm not sure what "behind the scenes" things
Cassandra does. I've received advice to avoid the built in secondary
indexes in Cassandra for some of these reasons. Also if row_loaded is
meant to implement some kind of queuing behavior, it could be the wrong
problem space for Cassandra as a result of all of the above.
On Sat, Mar 31, 2012 at 12:22 PM, aaron morton wrote:
> Is there anything in the logs when you run the queries ?
>
> Try turning the logging up to DEBUG on the node that fails to return and
> see what happens. You will see it send messages to other nodes and do work
> itself.
>
> One thing to note, a query that uses secondary indexes runs on a node for
> each token range. So it will use more than CL number of nodes.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 30/03/2012, at 11:52 AM, Chris Hart wrote:
>
> Hi,
>
> I have the following cluster:
>
> 136112946768375385385349842972707284580
> MountainViewRAC1 Up Normal 1.86 GB
> 20.00% 0
> MountainViewRAC1 Up Normal 2.17 GB
> 33.33% 56713727820156410577229101238628035242
> MountainViewRAC1 Up Normal 2.41 GB
> 33.33% 113427455640312821154458202477256070485
> Rackspace RAC1 Up Normal 3.9 GB
> 13.33% 136112946768375385385349842972707284580
>
> The following query runs quickly on all nodes except 1 MountainView node:
>
> select * from Access_Log where row_loaded = 0 limit 1;
>
> There is a secondary index on row_loaded. The query usually doesn't
> complete (but sometimes does) on the bad node and returns very quickly on
> all other nodes. I've upping the rpc timeout to a full minute
> (rpc_timeout_in_ms: 60000) in the yaml, but it still often doesn't complete
> in a minute. It seems just as likely to complete and takes about the same
> amount of time whether the limit is 1, 100 or 1000.
>
>
> Thanks for any help,
> Chris
>
>
>
--90e6ba6e8ef2e65af504bcb42ab0
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

This is all very hypothetical, but I've been bitten by this before=
.

Does row_loaded happen to be a binary or boolean valu=
e? =A0If so the secondary index generated by Cassandra will have at most 2 =
rows, and they'll be REALLY wide if you have a lot of entries. =A0Since=
Cassandra doesn't distribute columns over rows, those potentially very=
wide index rows, and their replicas, must live in SSTables in their entire=
ty on the nodes that own them (and their replicas).

Even though you limit 1, I'm not sure what "behind =
the scenes" things Cassandra does. =A0I've received advice to avoi=
d the built in secondary indexes in Cassandra for some of these reasons. =
=A0Also if row_loaded is meant to implement some kind of queuing behavior, =
it could be the wrong problem space for Cassandra as a result of all of the=
above.

The following query runs q=
uickly on all nodes except 1 MountainView node:

select * from Acces=
s_Log where row_loaded =3D 0 limit 1;

There is a secondary index on row_loaded. =A0The query usually doesn=
9;t complete (but sometimes does) on the bad node and returns very quickly =
on all other nodes. =A0I've upping the rpc timeout to a full minute (rp=
c_timeout_in_ms: 60000) in the yaml, but it still often doesn't complet=
e in a minute. =A0It seems just as likely to complete and takes about the s=
ame amount of time whether the limit is 1, 100 or 1000.