To some extent, the boot-strapping problem will be an issue with most
solutions: the data has to be duplicated from somewhere. Bootstrapping
should not cause much performance degradation unless you are already pushing
capacity limits. It's the decommissioning problem which makes Cassandra
somewhat problematic in your case. You grow your cluster x5 then write to
it. You have to perform a proper decommission when shrinking the cluster
again which involves validating and streaming data to the remaining
replicas: a fairly serious operation with TBs of data. For most realistic
situations, unless the cluster is completely read-only, you cant just kill
most of the nodes in the cluster.
I cant really think of a good, general, way to do this with just Cassandra
although there may be some hacktastical possibilities. I think a more
statically sized Cassandra cluster then a variable cache layer (memcached or
similar) is probably a better solution. This option kind of falls apart at
the terabytes of data range.
Have you considered using S3, Amazon cloud front or some other CDN instead
of rolling your own solution? For immutable data, its what they excel at.
Cassandra has amazing write capacity and its design focus is on scaling
writes. I would not really consider it a good tool for the job of serving
massive amounts of static content.
Dan
-----Original Message-----
From: Shaun Cutts [mailto:shaun@cuttshome.net]
Sent: March-03-11 13:00
To: user@cassandra.apache.org
Subject: question about replicas & dynamic response to load
Hello,
In our project our usage pattern is likely to be quite variable -- high for
a a few days, then lower, etc could vary as much (or more) as 10x from peak
to "non-peak". Also, much of our data is immutable -- but there is a
considerable amount of it -- perhaps in the single digit TBs. Finally, we
are hosting with amazon.
I'm looking for advice on how to vary the number of nodes dynamically, in
order to reduce our hosting costs at non-peak times. I worry that just
adding "new" nodes in response to demand will make things worse -- at least
temporarily -- as the new node copies data to itself; then bringing it down
will also cause a degradation.
I'm wondering if it is possible to bring up exact copies of other nodes? Or
alternately to take down a populated node containing (only?) immutable data,
then bring it up again when the need arises?
Are there reference/reading materials(/blogs) concerning dynamically varying
number of nodes in response to demand?
Thanks!
-- Shaun
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.872 / Virus Database: 271.1.1/3479 - Release Date: 03/03/11
02:34:00