From user-return-28584-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Sep 5 17:33:59 2012
Return-Path:
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
by minotaur.apache.org (Postfix) with SMTP id 3A01AD4D9
for ; Wed, 5 Sep 2012 17:33:59 +0000 (UTC)
Received: (qmail 59548 invoked by uid 500); 5 Sep 2012 17:33:56 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 59507 invoked by uid 500); 5 Sep 2012 17:33:56 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 59498 invoked by uid 99); 5 Sep 2012 17:33:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 17:33:56 +0000
X-ASF-Spam-Status: No, hits=1.0 required=5.0
tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS,TRACKER_ID
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.82.249.147] (HELO mail29.messagelabs.com) (216.82.249.147)
by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 17:33:46 +0000
X-Env-Sender: Bryce.Godfrey@azaleos.com
X-Msg-Ref: server-3.tower-29.messagelabs.com!1346866402!24172118!1
X-Originating-IP: [96.31.162.44]
X-StarScan-Received:
X-StarScan-Version: 6.6.1.3; banners=-,-,-
X-VirusChecked: Checked
Received: (qmail 23221 invoked from network); 5 Sep 2012 17:33:23 -0000
Received: from unknown (HELO smtp.azaleos.net) (96.31.162.44)
by server-3.tower-29.messagelabs.com with AES128-SHA encrypted SMTP; 5 Sep 2012 17:33:23 -0000
Received: from FISH-MBX-02.azaleos.net ([fe80::4918:2597:a7be:1075]) by
FISH-HUB-01.azaleos.net ([fe80::ecdb:527f:9348:37e%11]) with mapi id
14.03.0083.000; Wed, 5 Sep 2012 10:33:22 -0700
From: Bryce Godfrey
To: "user@cassandra.apache.org"
Subject: RE: Expanding cluster to include a new DR datacenter
Thread-Topic: Expanding cluster to include a new DR datacenter
Thread-Index: Ac2ApIfFaOPYziBeTeqfUNwTaymK8QAapDIAABKmdlAANGtEMAAShFoAAAYy6sAAdtKxAAAFIMDQABOF5QAACnmiEP//1c6A//8y61D//jOKsIAE4E4A///BTAAAMUcagP/+AoPQ//Q9vWA=
Date: Wed, 5 Sep 2012 17:33:21 +0000
Message-ID: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E658395@FISH-MBX-02.azaleos.net>
References: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E637337@FISH-MBX-02.azaleos.net>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E637B3A@FISH-MBX-02.azaleos.net>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E63928D@FISH-MBX-02.azaleos.net>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E6397BB@FISH-MBX-02.azaleos.net>
<4054967D-7C34-4964-A0AB-E9A0E59A2237@thelastpickle.com>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E63B105@FISH-MBX-02.azaleos.net>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E63B493@FISH-MBX-02.azaleos.net>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E63BF90@FISH-MBX-02.azaleos.net>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E63CE13@FISH-MBX-02.azaleos.net>
<4161BDED-7E04-4E9D-953C-9048EC130954@thelastpickle.com>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E6528FE@FISH-MBX-0 2.azaleos.net>
<6231C2AA-BC32-44AF-9E0E-1AB3065A256D@thelastpickle.com>
<054A19B0E41FDD43A58D2CC1E8C3FF5B1E6544F4@FISH-MBX-02.azaleos.net>
In-Reply-To: <054A19B0E41FDD43A58D2CC1E8C3FF5B1E6544F4@FISH-MBX-02.azaleos.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.20.0.100]
Content-Type: multipart/alternative;
boundary="_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E658395FISHMBX02azaleo_"
MIME-Version: 1.0
--_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E658395FISHMBX02azaleo_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
I just wanted to raise this back up again, see if anyone can provide some h=
elp. Thanks.
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Friday, August 31, 2012 11:47 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter
Here is the log around the drop keyspace with debugging enabled.
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, August 29, 2012 10:22 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
Can you provide the server log ?
If you can turn the log level up to DEBUG that would be handy as well.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 30/08/2012, at 8:22 AM, Bryce Godfrey > wrote:
Well I tried to drop the keyspace, but it's still there. No errors in logs=
and Cassandra-cli showed the schema agreement after the command. I took a=
snapshot of the system keyspace first. Nothing is crashing in the clients=
yet either, still able to read/write to that keyspace.
[default@EBonding] drop keyspace EBonding;
2eb11095-b8a8-31cd-80c3-c748d32a4208
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown] use EBonding;
Authenticated to keyspace: EBonding
[default@EBonding] describe;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, August 29, 2012 2:36 AM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
It would be handy to work out what the corruption is. Could you snapshot th=
e system keyspace and store it somewhere, just incase we can look at it lat=
er ?
Is there a way I can confirm this
Errors in the client and/or the server log is the the traditional way.
go about cleaning up/restoring the proper schema?
If you need to get it back, and can handle the down time, the simple thing =
is drop the KS and re-create it. Remember to take a snapshot first. Drop ke=
yspace takes one but it's the sort of thing I would do myself.
Or you can try _try_ nodetool resetlocalschema. Without knowing what the e=
rror is it's hard to say if it would work.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey > wrote:
I believe what may be really going on is that my schema is in a bad or corr=
upt state. I also have one keyspace that I just cannot drop an existing co=
lumn family from even though it shows no errors.
So right now I was able to get 4 of my 6 keyspaces over to Network Topology=
strategy.
I think I got into this bad state after pointing Opscenter at this cluster =
for the first time, as it started throwing errors after that and crashed a =
couple of my nodes until I stopped it and its agents.
Is there a way I can confirm this or go about cleaning up/restoring the pro=
per schema?
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Tuesday, August 28, 2012 11:09 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter
So in an interesting turn of events, this works on my other 4 keyspaces but=
just not this 'EBonding' one which will not recognize the changes. I can =
probably get around this by dropping and re-creating this keyspace since it=
s uptime is not too important for us.
[default@AlertStats] describe AlertStats;
Keyspace: AlertStats:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrateg=
y
Durable Writes: true
Options: [Fisher:3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey > wrote:
Same results. I restarted the node also to see if it just wasn't picking u=
p the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name fr=
om properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's =
what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could =
just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey > wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy =3D 'SimpleStrategy'
and strategy_options =3D {replication_factor : 2}
and durable_writes =3D true;
This is the only thing I see in the system log at the time on all the nodes=
:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (lin=
e 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 seri=
alized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Wri=
ting Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 =
ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Com=
pleted flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_k=
eyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe=
?
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there=
?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/08/2012, at 6:53 PM, Bryce Godfrey > wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.=
8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey > wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopolog=
y and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy =3D 'org.apache.cassandra.locator.NetworkTo=
pologyStrategy'
... and strategy_options=3D{Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there somethi=
ng else that I need to do? I've exited and restarted Cassandra-cli and it =
still shows the SimpleStrategy for that keyspace. Other nodes show the sam=
e information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://ww=
w.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-=
to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey > wrote:
We are in the process of building out a new DR system in another Data Cente=
r, and we want to mirror our Cassandra environment to that DR. I have a co=
uple questions on the best way to do this after reading the documentation o=
n the Datastax website. We didn't initially plan for this to be a DR setup=
when first deployed a while ago due to budgeting, but now we need to. So =
I'm just trying to nail down the order of doing this as well as any potenti=
al issues.
For the nodes, we don't plan on querying the servers in this DR until we fa=
il over to this data center. We are going to have 5 similar nodes in the =
DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using=
LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I=
change the replication strategy after joining the new nodes in the DR to N=
etworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new=
nodes. For the strategy options, just list the first dc until the second =
is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSni=
tch going to cause any issues? Since its in the Cassandra.yaml file I assu=
me a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would natu=
rally put all nodes in a DC in the same rack, I recommend using PropertyFil=
eSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regard=
less; it's harder to accidentally mess up.) A rolling restart is required =
to pick up the change. Make sure to fill out cassandra-topology.properties=
first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax
--_000_054A19B0E41FDD43A58D2CC1E8C3FF5B1E658395FISHMBX02azaleo_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I just wanted to raise th=
is back up again, see if anyone can provide some help. Thanks.

Well I tried to drop the =
keyspace, but it’s still there. No errors in logs and Cassandra=
-cli showed the schema agreement after the command. I took a snapshot
of the system keyspace first. Nothing is crashing in the clients yet=
either, still able to read/write to that keyspace.

It would be handy to work out what the corruption is=
. Could you snapshot the system keyspace and store it somewhere, just incas=
e we can look at it later ?

Is there a way I can conf=
irm this

Errors in the client and/or the server log is the th=
e traditional way.

go about cleaning up/rest=
oring the proper schema?

If you need to get it back, and can handle the down =
time, the simple thing is drop the KS and re-create it. Remember to take a =
snapshot first. Drop keyspace takes one but it's the sort of thing I would =
do myself.

Or you can try _try_ nodetool resetlocalschema. &nbs=
p;Without knowing what the error is it's hard to say if it would work.&nbsp=
;

I believe what may be rea=
lly going on is that my schema is in a bad or corrupt state. I also h=
ave one keyspace that I just cannot drop an existing column family
from even though it shows no errors.

<=
/o:p>

So right now I was able t=
o get 4 of my 6 keyspaces over to Network Topology strategy.

<=
/p>

I think I got into this b=
ad state after pointing Opscenter at this cluster for the first time, as it=
started throwing errors after that and crashed a couple
of my nodes until I stopped it and its agents.

<=
/p>

Is there a way I can conf=
irm this or go about cleaning up/restoring the proper schema?

So in an interesting turn=
of events, this works on my other 4 keyspaces but just not this ‘EBo=
nding’ one which will not recognize the changes. I can probably
get around this by dropping and re-creating this keyspace since its uptime=
is not too important for us.

Then I do a describe and =
it still shows the old strategy. Is there something else that I need =
to do? I’ve exited and restarted Cassandra-cli and it still
shows the SimpleStrategy for that keyspace. Other nodes show the sam=
e information.

We are in the process of building out a new DR syste=
m in another Data Center, and we want to mirror our Cassandra environment t=
o that DR. I have a couple questions on the best way to do this after=
reading the documentation on the Datastax
website. We didn’t initially plan for this to be a DR setup wh=
en first deployed a while ago due to budgeting, but now we need to. S=
o I’m just trying to nail down the order of doing this as well as any=
potential issues.

For the nodes, we don’t plan on querying the s=
ervers in this DR until we fail over to this data center. We ar=
e going to have 5 similar nodes in the DR, should I join them into the ring=
at token+1?

Join them at token+10 just to leave a little space. Make sure you=
're using LOCAL_QUORUM for your queries instead of regular QUORUM.

All keyspaces are set to the replication strategy of=
SimpleStrategy. Can I change the replication strategy after joining =
the new nodes in the DR to NetworkTopologyStategy with the updated replicat=
ion factor for each dr?

Switch your keyspaces over to NetworkTopologyStrategy before adding the new=
nodes. For the strategy options, just list the first dc until the se=
cond is up (e.g. {main_dc: 3}).

Lastly, is changing snitch from default of SimpleSni=
tch to RackInferringSnitch going to cause any issues? Since its in th=
e Cassandra.yaml file I assume a rolling restart to pick up the value would=
be ok?

This is the first thing you'll want to do. Unless your node IPs would=
naturally put all nodes in a DC in the same rack, I recommend using Proper=
tyFileSnitch, explicitly using the same rack. (I tend to prefer PFSni=
tch regardless; it's harder to accidentally
mess up.) A rolling restart is required to pick up the change. =
Make sure to fill out cassandra-topology.properties first if using PFSnitc=
h.