HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as acomponent while HBase tries to create paths with .snapshot as a component.This leads to issues in HBase, and one of HDFS or HBase needs to give upthe .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6(quite recently) and it may not be too late to change HBase to use adifferent path component in an upcoming new release.

In HBase these path names are not user visible. If there is a deployment of0.94.6, one could provide a migration tool that renames .snapshot to.hbase-snapshot or something to be able to move to the Snapshot release ofHadoop. On the other hand, .snapshot in HDFS is a user visible name and isa convention that is used by many file systems. It's a matter offamiliarity with such path names that would help users in using HDFSsnapshots.

I am including the hdfs-dev in this email. Would appreciate if we couldwork together and come up with a solution.

> Hi,> This afternoon Huned ad I discovered an issue while playing with HBase> Snapshots on top of Hadoop's Snapshot branch (> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).>> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a> component while HBase tries to create paths with .snapshot as a component.> This leads to issues in HBase, and one of HDFS or HBase needs to give up> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6> (quite recently) and it may not be too late to change HBase to use a> different path component in an upcoming new release.>> In HBase these path names are not user visible. If there is a deployment of> 0.94.6, one could provide a migration tool that renames .snapshot to> .hbase-snapshot or something to be able to move to the Snapshot release of> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is> a convention that is used by many file systems. It's a matter of> familiarity with such path names that would help users in using HDFS> snapshots.>> I am including the hdfs-dev in this email. Would appreciate if we could> work together and come up with a solution.>> You can find sample output from hdfs command here:> http://pastebin.com/bBqR4Fvr>> Cheers>

-

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

It would be great that if Hbase could use a different name since, as you mentioned, ".sanpshot" is user visible in HDFS. It is used for specifying snapshot paths. For example, the path /foo/.snapshot/s0/bar is used for accessing /foo/bar in snapshot s0 of /foo. Thanks.Tsz-Wo

HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as acomponent while HBase tries to create paths with .snapshot as a component.This leads to issues in HBase, and one of HDFS or HBase needs to give upthe .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6(quite recently) and it may not be too late to change HBase to use adifferent path component in an upcoming new release.

In HBase these path names are not user visible. If there is a deployment of0.94.6, one could provide a migration tool that renames .snapshot to.hbase-snapshot or something to be able to move to the Snapshot release ofHadoop. On the other hand, .snapshot in HDFS is a user visible name and isa convention that is used by many file systems. It's a matter offamiliarity with such path names that would help users in using HDFSsnapshots.

I am including the hdfs-dev in this email. Would appreciate if we couldwork together and come up with a solution.

HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as acomponent while HBase tries to create paths with .snapshot as a component.This leads to issues in HBase, and one of HDFS or HBase needs to give upthe .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6(quite recently) and it may not be too late to change HBase to use adifferent path component in an upcoming new release.

In HBase these path names are not user visible. If there is a deployment of0.94.6, one could provide a migration tool that renames .snapshot to.hbase-snapshot or something to be able to move to the Snapshot release ofHadoop. On the other hand, .snapshot in HDFS is a user visible name and isa convention that is used by many file systems. It's a matter offamiliarity with such path names that would help users in using HDFSsnapshots.

I am including the hdfs-dev in this email. Would appreciate if we couldwork together and come up with a solution.

> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.>> -- Lars>>>> ________________________________> From: Ted Yu <[EMAIL PROTECTED]>> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]> Sent: Monday, April 15, 2013 7:13 PM> Subject: collision in the naming of '.snapshot' directory between hdfs> snapshot and hbase snapshot>>> Hi,> This afternoon Huned ad I discovered an issue while playing with HBase> Snapshots on top of Hadoop's Snapshot branch (> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).>> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a> component while HBase tries to create paths with .snapshot as a component.> This leads to issues in HBase, and one of HDFS or HBase needs to give up> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6> (quite recently) and it may not be too late to change HBase to use a> different path component in an upcoming new release.>> In HBase these path names are not user visible. If there is a deployment of> 0.94.6, one could provide a migration tool that renames .snapshot to> .hbase-snapshot or something to be able to move to the Snapshot release of> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is> a convention that is used by many file systems. It's a matter of> familiarity with such path names that would help users in using HDFS> snapshots.>> I am including the hdfs-dev in this email. Would appreciate if we could> work together and come up with a solution.>> You can find sample output from hdfs command here:> http://pastebin.com/bBqR4Fvr>> Cheers>

-

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

> Lars:> I will go ahead and log an HBase JIRA, tomorrow morning.>> This hopefully would give people enough time to respond.>> Cheers>>> On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:>>> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.>>>> -- Lars>>>>>>>> ________________________________>> From: Ted Yu <[EMAIL PROTECTED]>>> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]>> Sent: Monday, April 15, 2013 7:13 PM>> Subject: collision in the naming of '.snapshot' directory between hdfs>> snapshot and hbase snapshot>>>>>> Hi,>> This afternoon Huned ad I discovered an issue while playing with HBase>> Snapshots on top of Hadoop's Snapshot branch (>> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).>>>> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a>> component while HBase tries to create paths with .snapshot as a component.>> This leads to issues in HBase, and one of HDFS or HBase needs to give up>> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6>> (quite recently) and it may not be too late to change HBase to use a>> different path component in an upcoming new release.>>>> In HBase these path names are not user visible. If there is a deployment>> of>> 0.94.6, one could provide a migration tool that renames .snapshot to>> .hbase-snapshot or something to be able to move to the Snapshot release of>> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is>> a convention that is used by many file systems. It's a matter of>> familiarity with such path names that would help users in using HDFS>> snapshots.>>>> I am including the hdfs-dev in this email. Would appreciate if we could>> work together and come up with a solution.>>>> You can find sample output from hdfs command here:>> http://pastebin.com/bBqR4Fvr>>>> Cheers>>>>

-

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

> I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot">> Please suggest better name in future correspondence.>> Thanks>> On Mon, Apr 15, 2013 at 8:12 PM, Ted Yu <[EMAIL PROTECTED]> wrote:>> > Lars:> > I will go ahead and log an HBase JIRA, tomorrow morning.> >> > This hopefully would give people enough time to respond.> >> > Cheers> >> >> > On Mon, Apr 15, 2013 at 8:00 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:> >> >> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.> >>> >> -- Lars> >>> >>> >>> >> ________________________________> >> From: Ted Yu <[EMAIL PROTECTED]>> >> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]> >> Sent: Monday, April 15, 2013 7:13 PM> >> Subject: collision in the naming of '.snapshot' directory between hdfs> >> snapshot and hbase snapshot> >>> >>> >> Hi,> >> This afternoon Huned ad I discovered an issue while playing with HBase> >> Snapshots on top of Hadoop's Snapshot branch (> >> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).> >>> >> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot> as a> >> component while HBase tries to create paths with .snapshot as a> component.> >> This leads to issues in HBase, and one of HDFS or HBase needs to give up> >> the .snapshot reserved keyword. HBase released Snapshots feature in> 0.94.6> >> (quite recently) and it may not be too late to change HBase to use a> >> different path component in an upcoming new release.> >>> >> In HBase these path names are not user visible. If there is a deployment> >> of> >> 0.94.6, one could provide a migration tool that renames .snapshot to> >> .hbase-snapshot or something to be able to move to the Snapshot release> of> >> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and> is> >> a convention that is used by many file systems. It's a matter of> >> familiarity with such path names that would help users in using HDFS> >> snapshots.> >>> >> I am including the hdfs-dev in this email. Would appreciate if we could> >> work together and come up with a solution.> >>> >> You can find sample output from hdfs command here:> >> http://pastebin.com/bBqR4Fvr> >>> >> Cheers> >>> >> >>

-

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

> constraints:>> 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.> 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will ever> make it to 1.x. This hdfs feature ideally this shouldn't affect current A> pache Hbase 0.94.x's.> 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions should> pick a different table snapshot name to respect fs conventions.>> proposed actions:>> 1) let's make the hbase snapshot for a conf variable. (hbase.> snapshots.dir) let's change the default for hbase 95+. (maybe> .hbase-snapshots). we'll also port this patch to 0.94.x> 2) let's publish instructions on how to update the hbase snapshot dir:> shutdown hbase, config update, rename dir, restart hbase.> 3) I lean towards leaving the current default hbase snapshot dir in 94> since it shouldn't be affected. upgrading hbase to 95/96 will require> shutdown and update scripts so it seems like the ideal time to autoforce> this default change.>> Thoughts?>>> On Monday, April 15, 2013, lars hofhansl wrote:>> > OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.> >> > -- Lars> >> >> >> > ________________________________> > From: Ted Yu <[EMAIL PROTECTED]>> > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]> > Sent: Monday, April 15, 2013 7:13 PM> > Subject: collision in the naming of '.snapshot' directory between hdfs> > snapshot and hbase snapshot> >> >> > Hi,> > This afternoon Huned ad I discovered an issue while playing with HBase> > Snapshots on top of Hadoop's Snapshot branch (> > http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).> >> > HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as> a> > component while HBase tries to create paths with .snapshot as a> component.> > This leads to issues in HBase, and one of HDFS or HBase needs to give up> > the .snapshot reserved keyword. HBase released Snapshots feature in> 0.94.6> > (quite recently) and it may not be too late to change HBase to use a> > different path component in an upcoming new release.> >> > In HBase these path names are not user visible. If there is a deployment> of> > 0.94.6, one could provide a migration tool that renames .snapshot to> > .hbase-snapshot or something to be able to move to the Snapshot release> of> > Hadoop. On the other hand, .snapshot in HDFS is a user visible name and> is> > a convention that is used by many file systems. It's a matter of> > familiarity with such path names that would help users in using HDFS> > snapshots.> >> > I am including the hdfs-dev in this email. Would appreciate if we could> > work together and come up with a solution.> >> > You can find sample output from hdfs command here:> > http://pastebin.com/bBqR4Fvr> >> > Cheers>>>> --> // Jonathan Hsieh (shay)> // Software Engineer, Cloudera> // [EMAIL PROTECTED]>

-

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

1) hbase 0.94.6 is released and .snapshot is hardcoded in there.2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it will evermake it to 1.x. This hdfs feature ideally this shouldn't affect current Apache Hbase 0.94.x's.3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions shouldpick a different table snapshot name to respect fs conventions.

proposed actions:

1) let's make the hbase snapshot for a conf variable. (hbase.snapshots.dir) let's change the default for hbase 95+. (maybe.hbase-snapshots). we'll also port this patch to 0.94.x2) let's publish instructions on how to update the hbase snapshot dir:shutdown hbase, config update, rename dir, restart hbase.3) I lean towards leaving the current default hbase snapshot dir in 94since it shouldn't be affected. upgrading hbase to 95/96 will requireshutdown and update scripts so it seems like the ideal time to autoforcethis default change.

Thoughts?On Monday, April 15, 2013, lars hofhansl wrote:

> OK. Let's try to fix that quickly, so that I can release HBase 0.94.7.>> -- Lars>>>> ________________________________> From: Ted Yu <[EMAIL PROTECTED]>> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]> Sent: Monday, April 15, 2013 7:13 PM> Subject: collision in the naming of '.snapshot' directory between hdfs> snapshot and hbase snapshot>>> Hi,> This afternoon Huned ad I discovered an issue while playing with HBase> Snapshots on top of Hadoop's Snapshot branch (> http://svn.apache.org/viewvc/hadoop/common/branches/HDFS-2802/).>> HDFS (built from HDFS-2802 branch) doesn't allow paths with .snapshot as a> component while HBase tries to create paths with .snapshot as a component.> This leads to issues in HBase, and one of HDFS or HBase needs to give up> the .snapshot reserved keyword. HBase released Snapshots feature in 0.94.6> (quite recently) and it may not be too late to change HBase to use a> different path component in an upcoming new release.>> In HBase these path names are not user visible. If there is a deployment of> 0.94.6, one could provide a migration tool that renames .snapshot to> .hbase-snapshot or something to be able to move to the Snapshot release of> Hadoop. On the other hand, .snapshot in HDFS is a user visible name and is> a convention that is used by many file systems. It's a matter of> familiarity with such path names that would help users in using HDFS> snapshots.>> I am including the hdfs-dev in this email. Would appreciate if we could> work together and come up with a solution.>> You can find sample output from hdfs command here:> http://pastebin.com/bBqR4Fvr>> Cheers

Because HDFS exposes the snapshots so that the normal file systemoperations are mapped inside snapshot dirs, I think HDFS reserving the.snapshot name makes sense. OTOH, nothing is specific about the dir namethat is chosen by HBase.

I would prefer to change the dir name in 0.94 as well, since 0.94 is alsobeing run on top of hadoop 2. Alternatively, we can detect the underlyingHadoop version, and use either .snapshot or .hbase_snapshot in 0.94depending on h1 & h2.

Pardon my late inquisition here but since HBase already shipped outwith a name .snapshots/, why do we force them to change it, and notrename HDFS' snapshots to use .hdfs-snapshots, given that HDFSSnapshots has not been released for any users yet. The way I see it,that'd be much more easier to do than making a workaround for a donedeal on HBase, which already has its snapshot's users.

@Tsz-Wo - If the snapshots in HDFS aren't a 'generic' featureapplicable to other FileSystem interface implementations as well, then.hdfs-snapshots should be fine for it - no?

On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <[EMAIL PROTECTED]> wrote:> Hi,> Please take a look at patch v5 attached to HBASE-8352.>> It would be nice to resolve this blocker today so that 0.94.7 RC can be cut.>> Thanks>> On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:>>> Please see my last comment on the jira. We can make this work without>> breaking users who are using HDFS snapshots.>>>> ------------------------------>> *From:* Ted Yu <[EMAIL PROTECTED]>>> *To:* [EMAIL PROTECTED]>> *Cc:* [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>>> *Sent:* Tuesday, April 16, 2013 10:00 AM>> *Subject:* Re: collision in the naming of '.snapshot' directory between>> hdfs snapshot and hbase snapshot>>>> Let's get proper release notes for HBASE-8352 .>>>> Either Lars or I can send out notification to user mailing list so that>> there is enough preparation for this change.>>>> Cheers>>>> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <[EMAIL PROTECTED]> wrote:>>>> I was away from keyboard when I asserted that hdfs snapshot was a hadoop>> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5 feature.>> (I'm a little surprised -- expected this to be a hadoop2 compat breaking>> feature) -- so I agree that this is a bit more urgent.>>>> Anyway, I agree that the fs .snapshot naming convention is long standing>> and should win.>>>> My concern is with breaking compatibility in 0.94 again -- if we don't go>> down the conf variable route, I consider having docs to properly document>> how to do the upgrade and caveats of doing the upgrade in the docs/release>> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to>> 0.94.7, and to possibly to 0.95).>>>> Jon.>>>> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <[EMAIL PROTECTED]> wrote:>>>> > bq. Alternatively, we can detect the underlying Hadoop version, and use>> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.>> >>> > I think this would introduce more confusion, especially for operations.>> >>> > Cheers>> >>> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <[EMAIL PROTECTED]>>> wrote:>> >>> > > Because HDFS exposes the snapshots so that the normal file system>> > > operations are mapped inside snapshot dirs, I think HDFS reserving the>> > > .snapshot name makes sense. OTOH, nothing is specific about the dir>> name>> > > that is chosen by HBase.>> > >>> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is>> also>> > > being run on top of hadoop 2. Alternatively, we can detect the>> underlying>> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94>> > > depending on h1 & h2.>> > >>> > > Enis>> > >>> > >>> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu <[EMAIL PROTECTED]> wrote:>> > >>> > > > bq. let's make the hbase snapshot for a conf variable.>> > > >>> > > > Once we decide on the new name of snapshot directory, we should still>> > use>> > > > hardcoded value. This aligns with current code base:>> > > > See this snippet from HConstants:>> > > >>> > > > public static final List<String> HBASE_NON_TABLE_DIRS >> > > >>> > > > Collections.unmodifiableList(Arrays.asList(new String[] {>> > > > HREGION_LOGDIR_NAME,>> > > >>> > > > HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,>> > > >>> > > > HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,

Harsh J

-

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

@HBase-dev, thanks for yielding the reserved word ".snapshot" to HDFS and the fast fix for addressing the problem. You guys have done a great job!@Harsh, it seems that more people think that .snapshot is better.Tsz-Wo

Thanks for the consideration but we've just committed a change to addressthis as HBASE-8352On Wednesday, April 17, 2013, Harsh J wrote:

> Pardon my late inquisition here but since HBase already shipped out> with a name .snapshots/, why do we force them to change it, and not> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS> Snapshots has not been released for any users yet. The way I see it,> that'd be much more easier to do than making a workaround for a done> deal on HBase, which already has its snapshot's users.>> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature> applicable to other FileSystem interface implementations as well, then> .hdfs-snapshots should be fine for it - no?>> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu <[EMAIL PROTECTED]> wrote:> > Hi,> > Please take a look at patch v5 attached to HBASE-8352.> >> > It would be nice to resolve this blocker today so that 0.94.7 RC can be> cut.> >> > Thanks> >> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl <[EMAIL PROTECTED]>> wrote:> >> >> Please see my last comment on the jira. We can make this work without> >> breaking users who are using HDFS snapshots.> >>> >> ------------------------------> >> *From:* Ted Yu <[EMAIL PROTECTED]>> >> *To:* [EMAIL PROTECTED]> >> *Cc:* [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>> >> *Sent:* Tuesday, April 16, 2013 10:00 AM> >> *Subject:* Re: collision in the naming of '.snapshot' directory between> >> hdfs snapshot and hbase snapshot> >>> >> Let's get proper release notes for HBASE-8352 .> >>> >> Either Lars or I can send out notification to user mailing list so that> >> there is enough preparation for this change.> >>> >> Cheers> >>> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh <[EMAIL PROTECTED]>> wrote:> >>> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop> >> 2.1 or 3.0 feature. Apparently it is targeted as a hadoop 2.0.5> feature.> >> (I'm a little surprised -- expected this to be a hadoop2 compat> breaking> >> feature) -- so I agree that this is a bit more urgent.> >>> >> Anyway, I agree that the fs .snapshot naming convention is long standing> >> and should win.> >>> >> My concern is with breaking compatibility in 0.94 again -- if we don't> go> >> down the conf variable route, I consider having docs to properly> document> >> how to do the upgrade and caveats of doing the upgrade in the> docs/release> >> notes blocker to hbase 0.94.7. (specifically mentioning from 0.94.6 to> >> 0.94.7, and to possibly to 0.95).> >>> >> Jon.> >>> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu <[EMAIL PROTECTED]> wrote:> >>> >> > bq. Alternatively, we can detect the underlying Hadoop version, and> use> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.> >> >> >> > I think this would introduce more confusion, especially for> operations.> >> >> >> > Cheers> >> >> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar <[EMAIL PROTECTED]>> >> wrote:> >> >> >> > > Because HDFS exposes the snapshots so that the normal file system> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving> the> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir> >> name> >> > > that is chosen by HBase.> >> > >> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is> >