Cloudera Hat: We are customer driven when it comes to features andthis is oft requested. 0.96 is a compatiblity breaking release and wehave some constraints there. Snapshots is mostly an additive featureso it is technically possible with minor compatibility concerns.

Apache Hat: Keeping features to new versions makes the most sense - itkeeps stable versions stable and encourages folks to move to newershinier versions. :). Ideally, with a healthy Apache project thatreleases regularly, the release schedule and feature set ofdistributions shouldn't affect the natural release cadence and featureset of the apache project.

At the moment the best guess for when 0.96 gets released is unknown.There will be a non-trivial amount of time necessary to hardensnapshots as well as all the other additions to that version. Itspretty plain to see that Cloudera and HWX are putting significantefforts into readying 0.96 as well.

Jon.

On Thu, Jan 10, 2013 at 9:36 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:> Oh, I meant the 1.0.x, 1.1.x, 2.x.x, etc version. Yeah, the -beta is not a good idea (IMHO).>>> I have to ask the Cloudera and Hortonworks folks then: Why not wait until 0.96 is stable? Why backport snapshots to 0.94?>> -- Lars>>>> ________________________________> From: Stack <[EMAIL PROTECTED]>> To: HBase Dev List <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>> Sent: Thursday, January 10, 2013 9:19 PM> Subject: Re: Upcoming merge of snapshots branch into trunk. (HBASE-6055 and HABSE-7290)>>> On Thu, Jan 10, 2013 at 7:34 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:>> Eventually we should switch to semantic versioning (like Hadoop).>>>>>> The -beta stuff? Nah, at least in Hadoop, it has been arbitrarily applied (and contended). Lets not use Hadoop as an example. We have some precedent for linux-y odd is unstable, even is stable. Lets hold to it I'd say.>>> It also depends on the timing of 0.96.>>The fact that two companies want to port this to 0.94 seems to indicate low confidence that we can ship a stable 0.96 soon.>>>>>> I think it is more that 0.96.0 is a singularity. Including 0.96 in a downstreamer's bundle only makes sense when the vendor is moving to a new major version. These major versions happen on a less frequent cycle. We just need to make sure 0.96 is out and well-baked the next time these cycles come around.>> St.Ack

Intel will be putting effort into stabilizing 0.96 for production use too.

On Friday, January 11, 2013, Jonathan Hsieh wrote:

> Cloudera Hat: We are customer driven when it comes to features and> this is oft requested. 0.96 is a compatiblity breaking release and we> have some constraints there. Snapshots is mostly an additive feature> so it is technically possible with minor compatibility concerns.>> Apache Hat: Keeping features to new versions makes the most sense - it> keeps stable versions stable and encourages folks to move to newer> shinier versions. :). Ideally, with a healthy Apache project that> releases regularly, the release schedule and feature set of> distributions shouldn't affect the natural release cadence and feature> set of the apache project.>> At the moment the best guess for when 0.96 gets released is unknown.> There will be a non-trivial amount of time necessary to harden> snapshots as well as all the other additions to that version. Its> pretty plain to see that Cloudera and HWX are putting significant> efforts into readying 0.96 as well.>> Jon.>> On Thu, Jan 10, 2013 at 9:36 PM, lars hofhansl <[EMAIL PROTECTED]<javascript:;>>> wrote:> > Oh, I meant the 1.0.x, 1.1.x, 2.x.x, etc version. Yeah, the -beta is not> a good idea (IMHO).> >> >> > I have to ask the Cloudera and Hortonworks folks then: Why not wait> until 0.96 is stable? Why backport snapshots to 0.94?> >> > -- Lars> >> >> >> > ________________________________> > From: Stack <[EMAIL PROTECTED] <javascript:;>>> > To: HBase Dev List <[EMAIL PROTECTED] <javascript:;>>; lars hofhansl> <[EMAIL PROTECTED] <javascript:;>>> > Sent: Thursday, January 10, 2013 9:19 PM> > Subject: Re: Upcoming merge of snapshots branch into trunk. (HBASE-6055> and HABSE-7290)> >> >> > On Thu, Jan 10, 2013 at 7:34 PM, lars hofhansl <[EMAIL PROTECTED]<javascript:;>>> wrote:> >> > Eventually we should switch to semantic versioning (like Hadoop).> >>> >>> >> > The -beta stuff? Nah, at least in Hadoop, it has been arbitrarily> applied (and contended). Lets not use Hadoop as an example. We have some> precedent for linux-y odd is unstable, even is stable. Lets hold to it I'd> say.> >> >> > It also depends on the timing of 0.96.> >>The fact that two companies want to port this to 0.94 seems to indicate> low confidence that we can ship a stable 0.96 soon.> >>> >>> >> > I think it is more that 0.96.0 is a singularity. Including 0.96 in a> downstreamer's bundle only makes sense when the vendor is moving to a new> major version. These major versions happen on a less frequent cycle. We> just need to make sure 0.96 is out and well-baked the next time these> cycles come around.> >> > St.Ack>>>> --> // Jonathan Hsieh (shay)> // Software Engineer, Cloudera> // [EMAIL PROTECTED] <javascript:;>>-- Best regards,

Matteo, Jesse and I seem to be getting to the point where have corefunctionality for offline snapshots (disable table, snapshot) and onlinesnapshot (snapshot an enabled table) committed, did another rev solidifyingfile layout, and have been steadily knocking off blocking and non-blockingfollow-on subtasks.

As a heads up, I think we are going to start considering a merge of thesnapshots branch to trunk. We agreed early on that we'd want 3 +1s beforethe merge occurred. Matteo, Jesse, and I worked on the core and allcommitters now so we could technically satisfy this, but I'd really like tohave at least 1 reviewer look at the whole thing who didn't write parts ofit. :) We'll also try to do another update on the design docs to help thereviews. Please consider taking a look at the branches and let us know.

We're going to be spending this week hardening and testing are reasonablescale to shake out more problems, so this may happen starting some timenext week.

Aside from asking for reviews, there are a few outstanding questions we'dlove to get your feedback on:HBASE-7471 - default configuration so that snapshots are available bydefault?HBASE-7360 - do we backport offline snapshots to the apache hbase 0.94 line?

So where is the code and jiras? Currently, there are two branches -- anoffline snapshot only branch (HBASE-6055) here (https://github.com/jyates/hbase/tree/snapshots akahttps://github.com/jmhsieh/hbase/tree/hbase-6055) and an offline + onlineone (HBASE-7290) here(https://github.com/jmhsieh/hbase/tree/snapshots). Currently thedifference between the two are 3-4 patches (they are fairly substantial butprimarily additive). We were being conservative initially and had hopedthat we could of done an offline merge earlier (and possibly merge with 94)and then a second round when the online snapshot was more robust. If ourtesting goes well this week, for trunk I'd lean more towards just doing onemerge pass with the offline+online branch.

Here's my current thoughts on the strategy and mechanics of the merge(hopefully sometime next week). Currently, our branches are based on a12/18 version of trunk. We're going to start doing merges of trunk into thesnapshot branch (to get all trunk patches into the snapshots). I'll alsopost a mega patch on review board after. After we get all the reviews, wecould go a couple of ways, and would like your thoughts:

1) just commit the consolidated mega patch to trunk. (every trunk stepshould compile, but we lose blame information)2) rebase each patch and then a fix up patch at the end (yuck -- may havebroken intermediate steps, but keeps blame info)3) create a branch in svn (we've been in github) and, after we do an svnmerge of the snapshot branch into trunk. (more work, but each commit ineach branch should compile, keep blame information).

#1 would look like this:

SS SS is a single mega patch commit. |S4 | A merge of trunk into the snapshot branch (could be multiple)| \|| T3 Trunk patch that when in while snapshots under dev.S3 || T2S2 |S1 | Snapshot branch patches \| T1 The 12/18 point where snapshots was last rebased. ...

#3 would be a bit of work but I think would be best in the end. I thinkthe history would roughly look like this: (S is a snapshot commit, T istrunk)

T4 Final merge commit into trunk. should be the same as S4 /|S4 | A merge of trunk into the snapshot branch (could be multiple)| \|| T3 Trunk patch that when in while snapshots under dev.S3 || T2S2 |S1 | Snapshot branch patches \| T1 The 12/18 point where snapshots was last rebased. ...

> Matteo, Jesse and I seem to be getting to the point where have core> functionality for offline snapshots (disable table, snapshot) and online> snapshot (snapshot an enabled table) committed, did another rev solidifying> file layout, and have been steadily knocking off blocking and non-blocking> follow-on subtasks.>>Hurray!

> Aside from asking for reviews, there are a few outstanding questions we'd> love to get your feedback on:> HBASE-7471 - default configuration so that snapshots are available by> default?>+1 on enabling by default in trunk/0.96

> HBASE-7360 - do we backport offline snapshots to the apache hbase 0.94> line?>>+0 tending toward -1. Patch is large for little benefit: i.e. offlinemerge (you have to take table offline, right?)

> So where is the code and jiras? Currently, there are two branches -- an> offline snapshot only branch (HBASE-6055) here (> https://github.com/jyates/hbase/tree/snapshots aka> https://github.com/jmhsieh/hbase/tree/hbase-6055) and an offline + online> one (HBASE-7290) here> (https://github.com/jmhsieh/hbase/tree/snapshots). Currently the> difference between the two are 3-4 patches (they are fairly substantial but> primarily additive). We were being conservative initially and had hoped> that we could of done an offline merge earlier (and possibly merge with 94)> and then a second round when the online snapshot was more robust. If our> testing goes well this week, for trunk I'd lean more towards just doing one> merge pass with the offline+online branch.>>So, you want us review over in the branches? Or do you have updatedpatches attached to 6055 and 7290? (Or you want us to wait till there isthe later reference mega-patch posted up on rb -- if rb will take it(smile)).

> 1) just commit the consolidated mega patch to trunk. (every trunk step> should compile, but we lose blame information)> 2) rebase each patch and then a fix up patch at the end (yuck -- may have> broken intermediate steps, but keeps blame info)> 3) create a branch in svn (we've been in github) and, after we do an svn> merge of the snapshot branch into trunk. (more work, but each commit in> each branch should compile, keep blame information).>>For #3, when you create a branch, it would be made with what is out ongithub after github had been brought current w/ trunk? Then, you wouldmerge in the svn branch into trunk? Isn't the commit history only going tobe one level deep? i.e. 'branch made from what was in github?" Or will ithave more than this?

> Hey Folks,>> Matteo, Jesse and I seem to be getting to the point where have core> functionality for offline snapshots (disable table, snapshot) and online> snapshot (snapshot an enabled table) committed, did another rev solidifying> file layout, and have been steadily knocking off blocking and non-blocking> follow-on subtasks.>> As a heads up, I think we are going to start considering a merge of the> snapshots branch to trunk. We agreed early on that we'd want 3 +1s before> the merge occurred. Matteo, Jesse, and I worked on the core and all> committers now so we could technically satisfy this, but I'd really like to> have at least 1 reviewer look at the whole thing who didn't write parts of> it. :) We'll also try to do another update on the design docs to help the> reviews. Please consider taking a look at the branches and let us know.>> We're going to be spending this week hardening and testing are reasonable> scale to shake out more problems, so this may happen starting some time> next week.>> Aside from asking for reviews, there are a few outstanding questions we'd> love to get your feedback on:> HBASE-7471 - default configuration so that snapshots are available by> default?> HBASE-7360 - do we backport offline snapshots to the apache hbase 0.94> line?>> So where is the code and jiras? Currently, there are two branches -- an> offline snapshot only branch (HBASE-6055) here (> https://github.com/jyates/hbase/tree/snapshots aka> https://github.com/jmhsieh/hbase/tree/hbase-6055) and an offline + online> one (HBASE-7290) here> (https://github.com/jmhsieh/hbase/tree/snapshots). Currently the> difference between the two are 3-4 patches (they are fairly substantial but> primarily additive). We were being conservative initially and had hoped> that we could of done an offline merge earlier (and possibly merge with 94)> and then a second round when the online snapshot was more robust. If our> testing goes well this week, for trunk I'd lean more towards just doing one> merge pass with the offline+online branch.>> Here's my current thoughts on the strategy and mechanics of the merge> (hopefully sometime next week). Currently, our branches are based on a> 12/18 version of trunk. We're going to start doing merges of trunk into the> snapshot branch (to get all trunk patches into the snapshots). I'll also> post a mega patch on review board after. After we get all the reviews, we> could go a couple of ways, and would like your thoughts:>> 1) just commit the consolidated mega patch to trunk. (every trunk step> should compile, but we lose blame information)> 2) rebase each patch and then a fix up patch at the end (yuck -- may have> broken intermediate steps, but keeps blame info)> 3) create a branch in svn (we've been in github) and, after we do an svn> merge of the snapshot branch into trunk. (more work, but each commit in> each branch should compile, keep blame information).>> #1 would look like this:>> SS SS is a single mega patch commit.

> Jon:> Merging option #3 looks attractive. Do you have estimate for how long trunk> should be frozen to other commits if we choose this approach ?>> The good news is that I don't think we actually need to freeze trunk. S4in the picture for #3 is something that can happen in parallel withoutfreezing trunk. If changes in trunk that breaks the code in snapshots(there currently are a few places), we'd do another S4-like commit (mergetrunk into snapshots, with review if non trivial) and then try to do the T4commit again. This is similar to the process used to merge various hdfs2ha-branches into hdfs trunk. Seemed like a good model to follow.

One catch is that we've been in git -- and since my svn-fu is weaker thangit-fu, I'd have to spend some time figuring out how to do this with svn (Iknow it is doable).> Is snapshot-work-0103 the latest branch for HBASE-7290 repo ?>> The snapshot-work-0103 branch is the latest stuff that has alot of extracode I've been keeping around in a compilable state, but not what I'dmerge. Not all of it will go into the online merge. Only the code in thejyates/snapshtos (offline) or jmhsieh/snapshots (offline+online) branch,which has been reviewed and accepted is what I'd consider merging. I'd betsome of the commits afterwards don't compile (the last few reviews requiredme to move code around between patches -- and I was only careful with whatwas going to be committed).> I ran Test*Snapshot* tests from this branch and found one failed test:>>> testValidateGlobalSnapshotDescriptor(org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils)> Time elapsed: 0.112 sec <<< ERROR!> com.google.protobuf.UninitializedMessageException: Message missing required> fields: name> at>> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)> at>> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription$Builder.build(HBaseProtos.java:11819)> at>> org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils.testValidateGlobalSnapshotDescriptor(TestSnapshotDescriptionUtils.java:108)>>This is safe to ignore for now -- I just intend to merge with the flushonline snapshot -- I don't intend to merge with the global online snapshotor logroll online snapshot. These last two would become full issues oncethe branch has been merged.> Cheers>> On Mon, Jan 7, 2013 at 10:07 AM, Jonathan Hsieh <[EMAIL PROTECTED]> wrote:>> > Hey Folks,> >> > Matteo, Jesse and I seem to be getting to the point where have core> > functionality for offline snapshots (disable table, snapshot) and online> > snapshot (snapshot an enabled table) committed, did another rev> solidifying> > file layout, and have been steadily knocking off blocking and> non-blocking> > follow-on subtasks.> >> > As a heads up, I think we are going to start considering a merge of the> > snapshots branch to trunk. We agreed early on that we'd want 3 +1s before> > the merge occurred. Matteo, Jesse, and I worked on the core and all> > committers now so we could technically satisfy this, but I'd really like> to> > have at least 1 reviewer look at the whole thing who didn't write parts> of> > it. :) We'll also try to do another update on the design docs to help> the> > reviews. Please consider taking a look at the branches and let us know.> >> > We're going to be spending this week hardening and testing are reasonable> > scale to shake out more problems, so this may happen starting some time> > next week.> >> > Aside from asking for reviews, there are a few outstanding questions we'd> > love to get your feedback on:> > HBASE-7471 - default configuration so that snapshots are available by> > default?> > HBASE-7360 - do we backport offline snapshots to the apache hbase 0.94> > line?> >> > So where is the code and jiras? Currently, there are two branches -- an