Activity

Can i have namespace in the specified HDFS folder?
Because in our multi-tenant scenarios, all the use resources are under the folder, files and hive tables and HBase datas.
We want to control all the data size in HDFS quota on the folder.

My propose is to have configurations for Namespace to modified the default path.

Peter Shi
added a comment - 09/Feb/15 07:22 Can i have namespace in the specified HDFS folder?
Because in our multi-tenant scenarios, all the use resources are under the folder, files and hive tables and HBase datas.
We want to control all the data size in HDFS quota on the folder.
My propose is to have configurations for Namespace to modified the default path.

Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?

Yes

How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?

As you would anticipate, eventually not, depending on how many tables each user creates and the capabilities of the namenode. I believe the intent of namespaces is to support multitenancy where a tenant shares a namespace and is a department or similar grouping within a larger organization. The namespace is a supergroup of tables sharing common ACLs, quotas, and the like.

I would like namespace to be totally logical separation all sharing the same region file. Is this case?

Andrew Purtell
added a comment - 20/Sep/14 22:21 Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?
Yes
How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?
As you would anticipate, eventually not, depending on how many tables each user creates and the capabilities of the namenode. I believe the intent of namespaces is to support multitenancy where a tenant shares a namespace and is a department or similar grouping within a larger organization. The namespace is a supergroup of tables sharing common ACLs, quotas, and the like.
I would like namespace to be totally logical separation all sharing the same region file. Is this case?
No it is not and this would be a substantial change.

Prasanna
added a comment - 20/Sep/14 21:36 Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?
How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?
I would like namespace to be totally logical separation all sharing the same region file. Is this case?
Thank you for any responses.

Enis Soztutar Where you at on your review? You want an update of the patch by Francis Liu? I committed HBASE-8778 because it looked like we needed a new revision. If current patch is good by you Enis Soztutar, I could back out HBASE-8778 because it would be easier applying that on top of ns.

stack
added a comment - 06/Aug/13 18:20 Enis Soztutar Where you at on your review? You want an update of the patch by Francis Liu ? I committed HBASE-8778 because it looked like we needed a new revision. If current patch is good by you Enis Soztutar , I could back out HBASE-8778 because it would be easier applying that on top of ns.

Francis Liu
added a comment - 01/Aug/13 16:53
The replaceAll() call only appears in this test. Can you tell me the reason ?
It was needed to test snapshot across namespace when we had '.' as the delimiter. I've removed replaceAll() in the new patch.
I think the above class was removed by HBASE-8764 . Please refresh your workspace and update your patch.
Re-removed it and a number of other files. I think it stayed during merge because I modified those files.
TestRestoreFlushSnapshotFromClient seems to hang:
Fixed this seemed to be a typo during cleanup.

Francis Liu
added a comment - 01/Aug/13 05:15
What if user already had table named 'default.x' ?
Then the path on hdfs will be <ROOT_DIR>/default/default.x
This is no longer an issue since we changed the delimiter to ':'. So the scenario where a table which already uses the delimiter no longer exists.
Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ?
Yep that image contains tables like that: ns1.foo and ns2.foo

2013-08-01 03:01:47,059 ERROR [RS_OPEN_REGION-kiyo:35035-0] handler.OpenRegionHandler(475): Failed open of region=clonedtb-1375326090526,c,1375326056050.2fffb17b7a45e6f20fd296ffe0b1a3e7., starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:692)
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:608)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:579)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4111)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4082)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4031)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3982)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:459)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:137)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:448)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:241)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3093)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:665)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:662)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]

Francis Liu
added a comment - 01/Aug/13 02:55 Yep it's attached to HBASE-8408 . Let me know if you're having trouble with it. The brief description is written in TestNamespaceUpgrade. See the source file for more info:
/**
Test upgrade from no namespace in 0.94 to namespace directory structure.
Mainly tests that tables are migrated and consistent. Also verifies
that snapshots have been migrated correctly.
*
Uses a tarball which is an image of an 0.94 hbase.rootdir.
*
Contains tables with currentKeys as the stored keys:
foo, ns1.foo, ns2.foo
*
Contains snapshots with snapshot
{num}
Keys as the contents:
snapshot1Keys, snapshot2Keys
*
*/

Ted Yu
added a comment - 01/Aug/13 02:44 Mind attaching the tar ball (src/test/data/TestNamespaceUpgrade.tgz) used by TestNamespaceUpgrade to the JIRA (with brief description on how it was generated) ?

Francis Liu
added a comment - 30/Jul/13 21:22
I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?)
Yep got that with your second question. Will do.
You updated rb?
Not yet. Will create a new RB once I've put in all the missing changes as you suggested.

stack
added a comment - 30/Jul/13 21:15 Are you suggesting I change it to <TableQualifier><...><...>-<NS>?
I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?)
Your comments on RB
You updated rb?

Francis Liu
added a comment - 30/Jul/13 18:40
Could use current link separator for added ns? hfilelink already has its own notion of separator?
Forgot to mention for HFileLink it uses hyphens and for backreferences it uses dots.

On 1., you mean existing HFileLinks? They would be to tables in default namespace?
Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name?

For migrated tables yes. But for new tables which use NS they could be to new NS tables so we need to add NS as part of the HFileLink. I've updated the HFileLink file format to look like <NS>,<TableQualifier><...><...> it will also recognize the older file format where '<NS>,' is missing (See HFileLink.java for more details). Are you suggesting I change it to <TableQualifier><...><...>-<NS>?

Could use current link separator for added ns? hfilelink already has its own notion of separator?

Yep, currently I'm using ','. I could use '' instead. So it becomes <NS><TableQualifier><...><...>?

On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)

Sorry I meant the HFileLink Back references, not region split references. I think it's used for quick reference counting to see if an archived HFile can be cleaned up or not.

3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)

Cool will add the check.

So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).

Yep, as far as I can tell. I've ran small, medium and large tests, it only had unrelated failures.

Francis Liu
added a comment - 30/Jul/13 18:24
On 1., you mean existing HFileLinks? They would be to tables in default namespace?
Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name?
For migrated tables yes. But for new tables which use NS they could be to new NS tables so we need to add NS as part of the HFileLink. I've updated the HFileLink file format to look like <NS>,<TableQualifier> <...> <...> it will also recognize the older file format where '<NS>,' is missing (See HFileLink.java for more details). Are you suggesting I change it to <TableQualifier> <...> <...>-<NS>?
Could use current link separator for added ns? hfilelink already has its own notion of separator?
Yep, currently I'm using ','. I could use ' ' instead. So it becomes <NS> <TableQualifier> <...> <...>?
On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)
Sorry I meant the HFileLink Back references, not region split references. I think it's used for quick reference counting to see if an archived HFile can be cleaned up or not.
3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)
Cool will add the check.
So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).
Yep, as far as I can tell. I've ran small, medium and large tests, it only had unrelated failures.
What is left to finish would you say?
Add a TableName PB
Your comments on RB
That's about it.

On 1., you mean existing HFileLinks? They would be to tables in default namespace? Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name? Could use current link separator for added ns? hfilelink already has its own notion of separator?

On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)

3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)

So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).

stack
added a comment - 28/Jul/13 05:52 On 1., you mean existing HFileLinks? They would be to tables in default namespace? Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name? Could use current link separator for added ns? hfilelink already has its own notion of separator?
On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)
3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)
So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).
Will review github.
What is left to finish would you say?

Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?

There's currently there are three places right now.

1. HFileLink
a. HFile references require include the table name as part of the filename
b. back references include table name as well
2. Snapshot names
a. In current behavior as Jesse Yates pointed out. Valid table names are valid snapshot names. Snapshots are stored on hdfs as a directory identified by their names. Though I'm inclined not to support ':' as a valid character for snapshot names so we can keep ':' as a special character?

Francis Liu
added a comment - 25/Jul/13 08:20
Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?
There's currently there are three places right now.
1. HFileLink
a. HFile references require include the table name as part of the filename
b. back references include table name as well
2. Snapshot names
a. In current behavior as Jesse Yates pointed out. Valid table names are valid snapshot names. Snapshots are stored on hdfs as a directory identified by their names. Though I'm inclined not to support ':' as a valid character for snapshot names so we can keep ':' as a special character?

stack
added a comment - 25/Jul/13 06:32 Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name
Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?
Presently snapshot creation will complain if it has a ':' in it
When you create a snapshot, can you use a TableName?

Enis Soztutar
added a comment - 25/Jul/13 02:32 I think above scheme is acceptable and the best we can do right now. ns:table also fits into cf:column naming as well.
rename FullyQualifiedTableName to TableName
makes sense.

Jesse Yates
added a comment - 24/Jul/13 20:06
snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it
It used to follow table name semantics, but looks like it just tries on the FS and fails if its not a valid character.

Francis Liu
added a comment - 24/Jul/13 13:50
A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)
Is meta public?

Francis Liu
added a comment - 24/Jul/13 13:42 I've pushed some changes into github addressing the following:
Change delimiter to ':'
Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name
changed filesystem table path to: <root>/data/<ns>/<table qualifier>
updated namespace and table name valid chars. namespace now only allows [A-Za-z0-9_]
rename FullyQualifiedTableName to TableName since all table names are fully qualified not unless explicitly stated
Some issues:
found out that ':' is not a valid character in HDFS (scratches head). so we'll really need an fs delimiter if we stick with this delimiter.
snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it
small and medium tests pass, didn't have time to run large yet
What do you guys think? I'll continue working on the other comments.
Check out the changes here:
https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_1

Enis Soztutar
added a comment - 23/Jul/13 07:16 Played with this for some time. Overall, shell commands, zookeeper data, ns table, hdfs data seems to be working. Did a skim through the commits at github. Waiting for the separator change.
A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)

Enis Soztutar
added a comment - 23/Jul/13 05:52 Just re-read your comments for not storing FQ name on file system. Then we can just do this for file link which already contains carefully-selected separator chars for different fields.

Enis Soztutar
added a comment - 23/Jul/13 05:50 Yes, we can use different separator chars for fqtn depending on context. META and other places will use colon, while filesystem might use some other char (possibly comma)

@enis it looks like ':' is not in this list of not-allowed characters in ZK. I naively tried it and it works. Just curious I'm not familiar with the HBase on windows effort. They won't be running HBase on HDFS? But on a windows DFS?

@stack I believe enis is suggesting on using a different delimiter when storing FQTN as part of filenames on hdfs.

Francis Liu
added a comment - 22/Jul/13 11:33 @enis it looks like ':' is not in this list of not-allowed characters in ZK. I naively tried it and it works. Just curious I'm not familiar with the HBase on windows effort. They won't be running HBase on HDFS? But on a windows DFS?
@stack I believe enis is suggesting on using a different delimiter when storing FQTN as part of filenames on hdfs.

Enis Soztutar
added a comment - 19/Jul/13 18:54 We can use a different separator for file system then we expose to the users. Haven't checked whether zk also allows ":". Using comma for this makes sense if you ask me.

Francis Liu
added a comment - 12/Jul/13 03:36 Forgot to mention, ':' is not a valid character on NTFS. So we'd have to have a different separate for full table names in HFileLinks. Possibly a switch to support this. Thoughts Enis Soztutar ?

I had a chat with Stack yesterday to iron out some design details. Here's a quick summary of things:

1. We will be going with ':' as the delimiter. We'll make it so that the tableNames stored on the filesystem don't get stored fully-qualified. So that leaves HFileLinks which needs table names to be written with the namespace.

2. a TableName POJO will be the class that gets passed around where a tableName reference is needed. HTable and HBaseAdmin will have the old apis overloaded with new apis that accept the TableName pojo. In addition, the old apis will recognize fully qualified table names. This keeps symmetry between both apis. Having colon as a delimiter will guarantee there is no confusion and clear intent.

3. Since we will be using a TableName pojo everywhere internally. That means all the coprocessor hooks that have tableNames will change as well. This will break backward compatibility.

4. External interfaces REST, Thrift and CLI will not require significant changes as tables can be referenced using fully-qualified. We will have to add namespace CRUD apis to REST and Thrift at a latter point.

Francis Liu
added a comment - 12/Jul/13 03:34 I had a chat with Stack yesterday to iron out some design details. Here's a quick summary of things:
1. We will be going with ':' as the delimiter. We'll make it so that the tableNames stored on the filesystem don't get stored fully-qualified. So that leaves HFileLinks which needs table names to be written with the namespace.
2. a TableName POJO will be the class that gets passed around where a tableName reference is needed. HTable and HBaseAdmin will have the old apis overloaded with new apis that accept the TableName pojo. In addition, the old apis will recognize fully qualified table names. This keeps symmetry between both apis. Having colon as a delimiter will guarantee there is no confusion and clear intent.
3. Since we will be using a TableName pojo everywhere internally. That means all the coprocessor hooks that have tableNames will change as well. This will break backward compatibility.
4. External interfaces REST, Thrift and CLI will not require significant changes as tables can be referenced using fully-qualified. We will have to add namespace CRUD apis to REST and Thrift at a latter point.
5. In a separate patch remove '.' prefix in non-table dirs in root dir

stack
added a comment - 09/Jul/13 07:16 I took a quick look over at https://github.com/francisliu/hbase_namespace/commit/18973af443e8c6424c978364e2d0db2a5abe1286#L5R705
Patch is massive but seems to be in the right direction. I'm thinking weeks – three to four weeks – for this to stabilize and get committed?

How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?

It depends on how we'd wanna deal with it. Presently it's addressable as all the apis recognize FQTN names with '.' as the delimiter. If we go this route there won't be much work, just some cleanup to use FullyQualifiedTableName were needed.

If we decide to have a different external and internal delimiter or overload the cli api there would be some amount of work.

Francis Liu
added a comment - 08/Jul/13 19:53
How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?
It depends on how we'd wanna deal with it. Presently it's addressable as all the apis recognize FQTN names with '.' as the delimiter. If we go this route there won't be much work, just some cleanup to use FullyQualifiedTableName were needed.
If we decide to have a different external and internal delimiter or overload the cli api there would be some amount of work.
In either case could be a separate patch.

stack
added a comment - 08/Jul/13 18:31 How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?
Yeah, HC/HCM should probably be evolving at least.
On the tableOrRegionName, yeah, should probably a utility that is called to figure out which and then once that is known, the appropriate api is called.
Not sure why the commons-io. Ditto on test failures. Yes, possible unrelated to your changes.
Let me look at your work over in git.

Stack I've pushed changes into github, the changes mainly addresses one thing: make namespace as first class. I've replaced internal apis which took "byte[] tableName" as a constant to use FullyQualifiedTableName instead. For HTable and HBaseAdmin, I've overloaded such functions instead to keep backward compatibility. Also created a new branch in the repo which is a rebase off trunk yesterday. I'll address your RB comments today.

Some issues:

The external interfaces: cli, thrift, rest and MR don't have the namespace as first class treatment yet

I initially went by the InterfaceAudience annotations but after talking to you guys it seems this is needs some updating as HConnection and HConnectionManager are both marked public tho they shouldn't be?

There are some apis (ie compact()) which have an overloaded parameter ie "byte[] tableOrRegionName" should we be splitting this into two apis and deprecate the old one?

Need to create a FullyQualifiedTableName PB equivalent and update PB rpc and messages as needed

I always needed to add commons-io as a maven dependency else things won't compile and these aren't related to my changes

Ran small,med,large tests and have these failures which aren't related to the patch: TestHCM.testClusterStatus, TestLogRolling.testCompactionRecordDoesntBlockRolling, TestShell

Francis Liu
added a comment - 08/Jul/13 17:38 Stack I've pushed changes into github, the changes mainly addresses one thing: make namespace as first class. I've replaced internal apis which took "byte[] tableName" as a constant to use FullyQualifiedTableName instead. For HTable and HBaseAdmin, I've overloaded such functions instead to keep backward compatibility. Also created a new branch in the repo which is a rebase off trunk yesterday. I'll address your RB comments today.
Some issues:
The external interfaces: cli, thrift, rest and MR don't have the namespace as first class treatment yet
I initially went by the InterfaceAudience annotations but after talking to you guys it seems this is needs some updating as HConnection and HConnectionManager are both marked public tho they shouldn't be?
There are some apis (ie compact()) which have an overloaded parameter ie "byte[] tableOrRegionName" should we be splitting this into two apis and deprecate the old one?
Need to create a FullyQualifiedTableName PB equivalent and update PB rpc and messages as needed
I always needed to add commons-io as a maven dependency else things won't compile and these aren't related to my changes
Ran small,med,large tests and have these failures which aren't related to the patch: TestHCM.testClusterStatus, TestLogRolling.testCompactionRecordDoesntBlockRolling, TestShell

Sergey Shelukhin
added a comment - 24/Jun/13 18:39 To solve ambiguity create should have for FQ overload imo. Then, that means you always operate on existing tables if they have a dot, so you can just prevent conflicts at create time.

'/' probably won't work. Or at least, it won't work up in zk because you want to have RS watch tables come and go (or is it NS come and go). If tables, you can't set a recursive watch so would seem to rule out '/'.

stack
added a comment - 22/Jun/13 21:44 '/' probably won't work. Or at least, it won't work up in zk because you want to have RS watch tables come and go (or is it NS come and go). If tables, you can't set a recursive watch so would seem to rule out '/'.

But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.

The argument above is that the current implementation has been deployed some where so the current implementation is what we have to commit?

Can we figure out something to bring along these legacy folks Francis? They will have to restart anyways going to the new stuff? Can we treat this problem apart from how ns goes into trunk/0.95?

Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?

How would that work? How would we distingush a FQTN from a plain TN if the TN has the delimiter in it as part of its name. I can see helper methods that the shell or command-line tools could use but don't see these being run on every tablename we are passed.

What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?

No conflation of namespace and tablename. Clean deliniation between the ns and tablename concepts. Users can take on the new namespace feature if they want to. No hackery.

Chatting today, the '/' character came up again. Did we rule this out as a delimiter? Yeah '/' is meaningful in ext and ntfs but can't we leverage their interpretation; e.g. ns is a directory of tables? Ditto up in zk.

stack
added a comment - 22/Jun/13 04:18 Sorry, meant overload.
But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.
The argument above is that the current implementation has been deployed some where so the current implementation is what we have to commit?
Can we figure out something to bring along these legacy folks Francis? They will have to restart anyways going to the new stuff? Can we treat this problem apart from how ns goes into trunk/0.95?
Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?
How would that work? How would we distingush a FQTN from a plain TN if the TN has the delimiter in it as part of its name. I can see helper methods that the shell or command-line tools could use but don't see these being run on every tablename we are passed.
What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?
No conflation of namespace and tablename. Clean deliniation between the ns and tablename concepts. Users can take on the new namespace feature if they want to. No hackery.
Chatting today, the '/' character came up again. Did we rule this out as a delimiter? Yeah '/' is meaningful in ext and ntfs but can't we leverage their interpretation; e.g. ns is a directory of tables? Ditto up in zk.

The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?

We are not married to the current implementation yet. But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.

Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?

What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?

*Nit side note: Shouldn't it be be called overload (not override), since we are planning on overloading a a method or am I thinking of one of the other approaches?

Francis Liu
added a comment - 21/Jun/13 18:19
The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?
We are not married to the current implementation yet. But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.
Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?
What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?
*Nit side note: Shouldn't it be be called overload (not override), since we are planning on overloading a a method or am I thinking of one of the other approaches?

But for the java apis we won't recognize an FQTN string in the public apis?

Well, there would be helper classes for making sense of the FQTN that could be used by shell, rest, etc., but, yeah, the thought would be that in the API, there is no namepacing, not unless you ask for it explicitly.

Let me look at the trick for how much we'd have to carry into the future.

stack
added a comment - 21/Jun/13 16:50 The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?
This is 0.96. Y! will have to do a 'migration' to go up on 0.96 anyways? Can you think of a 'step' that you could add for Y! cluster that would help here?
Your users are doing 'new HTable(conf, "ns.sometable");' and away they go?
But for the java apis we won't recognize an FQTN string in the public apis?
Well, there would be helper classes for making sense of the FQTN that could be used by shell, rest, etc., but, yeah, the thought would be that in the API, there is no namepacing, not unless you ask for it explicitly.
Let me look at the trick for how much we'd have to carry into the future.

The only dots we use are for namespaces and that's only in our sandbox so no worries there. My concern was that we'd have to ask users to upgrade their code in order to start using namespaces.

So we'll have an internal delimiter '.' and an external one for external tools such as cli, rest, etc. But for the java apis we won't recognize an FQTN string in the public apis?

I thought you had trick to make '.' work?

Yes, I do. Enis Soztutar pointed out that we will have to live with that trick in later versions. Just making sure you guys are ok with that. Have a look at it on github, just look at TableName.java that should help you get the gist of it.

Francis Liu
added a comment - 21/Jun/13 16:31
You mean Y! They are using '.' for separator currently?
The only dots we use are for namespaces and that's only in our sandbox so no worries there. My concern was that we'd have to ask users to upgrade their code in order to start using namespaces.
So we'll have an internal delimiter '.' and an external one for external tools such as cli, rest , etc. But for the java apis we won't recognize an FQTN string in the public apis?
I thought you had trick to make '.' work?
Yes, I do. Enis Soztutar pointed out that we will have to live with that trick in later versions. Just making sure you guys are ok with that. Have a look at it on github, just look at TableName.java that should help you get the gist of it.

One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS.

You mean Y! They are using '.' for separator currently?

As opposed to having just a table name then it's just a simple config change and restart.

No override of APIs?

We have a special KV comparator for .META. table. It leverages the ',' position as delimiter between table name and startcode in region name. I suppose we could change the KV comparator so it looks for ','.

stack
added a comment - 21/Jun/13 05:46 One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS.
You mean Y! They are using '.' for separator currently?
As opposed to having just a table name then it's just a simple config change and restart.
No override of APIs?
We have a special KV comparator for .META. table. It leverages the ',' position as delimiter between table name and startcode in region name. I suppose we could change the KV comparator so it looks for ','.
I thought you had trick to make '.' work?

Enis Soztutar
added a comment - 21/Jun/13 03:44 +thrift as well. That is why I am in favor of option 2 now.
',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?
I think we can use this. We are relying on it for partitioning region names, and in META. So it should be compatible with that framework.

Francis Liu
added a comment - 21/Jun/13 03:25
The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).
I see, so what delimiter are we planning on using? '^' and '@' isn't appealing. ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?

I think the old API will be against default NS.
The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

Is it possible to support FQTN as well? Or is going this route to avoid surprise completely? One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. As opposed to having just a table name then it's just a simple config change and restart.

Francis Liu
added a comment - 21/Jun/13 03:02
I think the old API will be against default NS.
The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).
Is it possible to support FQTN as well? Or is going this route to avoid surprise completely? One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. As opposed to having just a table name then it's just a simple config change and restart.

stack
added a comment - 21/Jun/13 00:39 If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?
I think the old API will be against default NS.
The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

One problem with option 4 is that we want to pay the price of migration only one between 0.94->0.96. If we do that, then it means we have to carry the exception tables code for all the releases going forward. Option 1 better than this I think? Note that surprise #1 also applies here as well.

Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

It should use the default ns. I think the idea is that there will not be a public facing thing called "fully qualified table name" in Elliot's approach. Although internally, we will need one, hence my tendency to go with option 2 over 3 (see my above comment): "namespace,table" seems good enough for me.

Enis Soztutar
added a comment - 21/Jun/13 00:06 One problem with option 4 is that we want to pay the price of migration only one between 0.94->0.96. If we do that, then it means we have to carry the exception tables code for all the releases going forward. Option 1 better than this I think? Note that surprise #1 also applies here as well.
Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?
It should use the default ns. I think the idea is that there will not be a public facing thing called "fully qualified table name" in Elliot's approach. Although internally, we will need one, hence my tendency to go with option 2 over 3 (see my above comment): "namespace,table" seems good enough for me.

Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.

The approach I proposed earlier would avoid having to do all the api stuff as part of the first namespace checkin as well as make use of '.' as a delimeter. The suprises are as I mentioned. We can incrementally add the apis.

Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

For some reason I'm not getting any jira or RB emails. Will take a look.

Francis Liu
added a comment - 20/Jun/13 21:42
Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.
The approach I proposed earlier would avoid having to do all the api stuff as part of the first namespace checkin as well as make use of '.' as a delimeter. The suprises are as I mentioned. We can incrementally add the apis.
Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?
For some reason I'm not getting any jira or RB emails. Will take a look.

Francis Liu There will be a migration evaluation script that will looks for presence of stuff like hfile v1s – they must be compacted away before you can upgrade – and this same step could check table names and if any w/ dot found, list options. This script would be run against 0.94 install before shutting down for upgrade (Yes two scripts, a checker, and then a doer).

Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.

stack
added a comment - 20/Jun/13 19:47 Francis Liu There will be a migration evaluation script that will looks for presence of stuff like hfile v1s – they must be compacted away before you can upgrade – and this same step could check table names and if any w/ dot found, list options. This script would be run against 0.94 install before shutting down for upgrade (Yes two scripts, a checker, and then a doer).
Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.
Did you get my review comments up on rb francis?
On dots in namespace, no, if it simplifies.

Sergey Shelukhin
added a comment - 20/Jun/13 19:38
exceptionNS.add(tableName.getNamespaceAsString());
What is the current thinking on dots in namespaces and names? Presumably one table could prevent the creation of multiple namespaces if dots are allowed in namespace name, which I thought they are

Francis Liu
added a comment - 20/Jun/13 19:11 Oops sorry I guess I'm talking about two scripts. One to check if some surprising migration needs to be done and provide links/options. And another that does the actual migration.

I thought of a way to implement Sergey Shelukhin idea. It was simple enough so I thought I'd give it a try. Essentially keep an in-memory list of tables which make use of the delimiter (ie '.') and consider these tables as exceptions to the namespace rule and handle them properly to make sure they are part of the default namespace. Have an added constraint that prevent creation of namespaces and tables that would conflict with any of the exception tables (ie ns1 and ns1.foo).

Surprise here is:

you can't create tables with the delimiter no longer unless you create the appropriate namespace.

you can't create tables/namespace which conflict the exception tables/namespaces

the exception list is derived by scanning the default namespace directories in .tmp, .data and .archive

Francis Liu
added a comment - 20/Jun/13 18:25 I thought of a way to implement Sergey Shelukhin idea. It was simple enough so I thought I'd give it a try. Essentially keep an in-memory list of tables which make use of the delimiter (ie '.') and consider these tables as exceptions to the namespace rule and handle them properly to make sure they are part of the default namespace. Have an added constraint that prevent creation of namespaces and tables that would conflict with any of the exception tables (ie ns1 and ns1.foo).
Surprise here is:
you can't create tables with the delimiter no longer unless you create the appropriate namespace.
you can't create tables/namespace which conflict the exception tables/namespaces
the exception list is derived by scanning the default namespace directories in .tmp, .data and .archive
Here's a sample of how it works. I've updated the TestNamespaceUpgrade test to verify that it works:
https://github.com/francisliu/hbase_namespace/tree/core_8408_exception_list

Stack, I'm leaning towards having the migration operation done manually by calling a script as well. Which options do we provide the user? Also it might be better if the script is portable enough that they can run on an existing 0.94 cluster so they don't have to find out during the actual upgrade process.

Francis Liu
added a comment - 20/Jun/13 18:23 Stack , I'm leaning towards having the migration operation done manually by calling a script as well. Which options do we provide the user? Also it might be better if the script is portable enough that they can run on an existing 0.94 cluster so they don't have to find out during the actual upgrade process.

I was thinking more along the lines of doing the non-surprising thing silently (if the user doesn't want to use namespaces he will never notice anything at all), and allowing the user to resolve conflicts as necessary while preventing them from unwittingly shooting himself in the foot; but migration-time options also work.

Sergey Shelukhin
added a comment - 19/Jun/13 01:03 I was thinking more along the lines of doing the non-surprising thing silently (if the user doesn't want to use namespaces he will never notice anything at all), and allowing the user to resolve conflicts as necessary while preventing them from unwittingly shooting himself in the foot; but migration-time options also work.

Option 4 would get us the '.' back (that would mean that we'd do options 3 and 4).

Migrating, we will have a pre-update step where you run a script and it will check your install for v1 hfiles AND if you have tables with dots in them. If you have dots in your table name, you will be pointed at a page that describes the options available to you.

stack
added a comment - 19/Jun/13 00:34 Option 4 would get us the '.' back (that would mean that we'd do options 3 and 4).
Migrating, we will have a pre-update step where you run a script and it will check your install for v1 hfiles AND if you have tables with dots in them. If you have dots in your table name, you will be pointed at a page that describes the options available to you.

Option 4 is to use a dot, not create any automatic namespaces, have well-defined name resolution rules, and prohibit creating conflicts under those (also prohibit any further dots in table names for good measure). E.g. if user has org.my.cool.table, org.my.foo.table they go under default, and namespaces org, org.my, org.my.cool, and org.my.cool2 cannot be created at all until this table is renamed (or maybe they can be created by explicitly specifying that all tables should be renamed/moved under namespace keeping essentially the same name given the dots).
That way it's explicit to the user what needs to be done (no shooting-in-the-foot); there are dots; and there's minimum surprise.
The extra dev work is for mapping such table names to HDFS paths/etc.

Sergey Shelukhin
added a comment - 19/Jun/13 00:01 Option 4 is to use a dot, not create any automatic namespaces, have well-defined name resolution rules, and prohibit creating conflicts under those (also prohibit any further dots in table names for good measure). E.g. if user has org.my.cool.table, org.my.foo.table they go under default, and namespaces org, org.my, org.my.cool, and org.my.cool2 cannot be created at all until this table is renamed (or maybe they can be created by explicitly specifying that all tables should be renamed/moved under namespace keeping essentially the same name given the dots).
That way it's explicit to the user what needs to be done (no shooting-in-the-foot); there are dots; and there's minimum surprise.
The extra dev work is for mapping such table names to HDFS paths/etc.

stack
added a comment - 18/Jun/13 22:37 If we use '@', won't we have to swap the order so it is table 'at' namespace? (Or, the '@' would seem to imply that).
Between 2 and 3, if we choose 3, can't we keep more hidden from the user – or allowing your 'external tools' argument, we should do 2 and 3?

BTW regarding separator chars:
"\/:*?<>|" NTFS does not allow these chars
"\/~" Meaningful in ext/ntfs.
"-" and "_" cannot be used because they are legal in table names.
"#" Comment in many SQL dialects, not compatible with URL encoding.
"$%&" Used in shell scripts

This leaves: "!@^+=," if we disregard parenthesis chars. These also seem to work for HDFS. "," is our internal separator for region names and META naming. @ also seems good enough. Haven't tried these for znodes.

It seems that we could not reach an agreement on which proposal to implement, where they are
1) use dot as separator, autogenerate namespaces on migration
2) use one of the candidate char's above as ns and table separator.
3) In all places that require a table name, override with accepting namespace.

However, it seems that we definitely have to have an internal separator char and a deterministic way of encoding <ns,table> pair into a string and decoding it back. This is because we need to do this for znodes in zktable, tablelockmanager, etc as well as META. So, I think it is a matter of whether we want to expose the separator char between proposal 2 and 3. I am in favor of doing 2, since it will allow better interop with external tools, and the code changes needed for 3 is significant.

Enis Soztutar
added a comment - 18/Jun/13 22:09 BTW regarding separator chars:
"\/:*?<>|" NTFS does not allow these chars
"\/~" Meaningful in ext/ntfs.
"-" and "_" cannot be used because they are legal in table names.
"#" Comment in many SQL dialects, not compatible with URL encoding.
"$%&" Used in shell scripts
This leaves: "!@^+=," if we disregard parenthesis chars. These also seem to work for HDFS. "," is our internal separator for region names and META naming. @ also seems good enough. Haven't tried these for znodes.
It seems that we could not reach an agreement on which proposal to implement, where they are
1) use dot as separator, autogenerate namespaces on migration
2) use one of the candidate char's above as ns and table separator.
3) In all places that require a table name, override with accepting namespace.
However, it seems that we definitely have to have an internal separator char and a deterministic way of encoding <ns,table> pair into a string and decoding it back. This is because we need to do this for znodes in zktable, tablelockmanager, etc as well as META. So, I think it is a matter of whether we want to expose the separator char between proposal 2 and 3. I am in favor of doing 2, since it will allow better interop with external tools, and the code changes needed for 3 is significant.

Francis Liu Are we going to implement the Elliott suggestion adding overrides for namespaces? I was chatting w/ Nick Dimiduk last weds and he made a comment that helped: he said that if a choice between devs doing more work to save users 'surprise', then the answer is clear.

Another thought is that we need to hurry up and get this in. I'll do a review now. Who else is reviewing?

stack
added a comment - 18/Jun/13 21:18 Francis Liu Are we going to implement the Elliott suggestion adding overrides for namespaces? I was chatting w/ Nick Dimiduk last weds and he made a comment that helped: he said that if a choice between devs doing more work to save users 'surprise', then the answer is clear.
Another thought is that we need to hurry up and get this in. I'll do a review now. Who else is reviewing?
Thanks Francis.

I am doing the review, but could definitely use more reviews. After the first patch (HBASE-8408) is +1'ed I will create a branch and commit there, since we will need the rest of the subtasks in shape before proposing the merge.

Enis Soztutar
added a comment - 07/Jun/13 21:24 I am doing the review, but could definitely use more reviews. After the first patch ( HBASE-8408 ) is +1'ed I will create a branch and commit there, since we will need the rest of the subtasks in shape before proposing the merge.

stack
added a comment - 07/Jun/13 20:27 What is state of this feature? What else needs to be done other than more review and test? Are we going to make a branch for it?
Francis Liu How much testing of this feature has there been? Thanks boss.

Francis Liu
added a comment - 13/May/13 21:19 I've uploaded a first stab of the migration code on RB and HBASE-8408 . Haven't addressed any of the comments yet so I could get this out first.
I added a unit test which tests migration on a tarballed 0.94 rootdir. It does simple verification that the tables and snapshots have been migrated properly.

It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.

No it shouldn't. There was no consensus on the list and this is an exploratory effort so no need of constraint.

One thought I had was that if all tables need to be moved, mayhaps we just say snapshots are broke on upgrade or rather than a tool for table rename, instead, we'd have a tool that did surgery on snapshots to adjust them to match the new layout (advantage of this latter is that it can be done offline, not inline with migration – though, I have no idea this would work... ask the snapshotters Jon, Matteo and Jesse).

stack
added a comment - 10/May/13 17:58 It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.
No it shouldn't. There was no consensus on the list and this is an exploratory effort so no need of constraint.
One thought I had was that if all tables need to be moved, mayhaps we just say snapshots are broke on upgrade or rather than a tool for table rename, instead, we'd have a tool that did surgery on snapshots to adjust them to match the new layout (advantage of this latter is that it can be done offline, not inline with migration – though, I have no idea this would work... ask the snapshotters Jon, Matteo and Jesse).

Ted Yu
added a comment - 10/May/13 16:47 a RenameTable tool will be provided to rename offline tables
It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.

Francis Liu
added a comment - 10/May/13 16:41 It seems we've come to a consensus on the upgrade approach.
In essence it'll do the following:
Tables without '.' will be moved into the default namespace.
Tables with '.' will be move into new namespaces
namespaces will be delimited by the last '.' in the table name
ie org.apache.hbase.MyTable, namespace = org.apache.hbase
In both cases the oldTableName is the same as the fullTableName
all existing apis and cli commands will be expecting full table names unless explicitly stated
a RenameTable tool will be provided to rename offline tables
So I'll start implementing it.
Discussion is here:
http://grokbase.com/p/hbase/dev/13598jegtp/discuss-namespace-delimiter

Enis Soztutar
added a comment - 09/May/13 22:28 Enis Soztutar do you have any bandwidth? Since you've shown interested earlier
I am up for it.
Would a quick meetup with the people involved help iron things out?
+1. It is great that 0.94 version is in shape. The remaining questions are to handle migration, and the separator issue, and testing I guess.
Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.
I am not a big fan of svn branches, but if it will help in reviews and testing, I can help with that.

Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

Our 0.94 patch is almost fully-baked so far so good...we already have it running in sandbox with users. Would a quick meetup with the people involved help iron things out?

Francis Liu
added a comment - 09/May/13 01:05
Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).
Our 0.94 patch is almost fully-baked so far so good...we already have it running in sandbox with users. Would a quick meetup with the people involved help iron things out?

Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

stack
added a comment - 09/May/13 00:22 Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

Are we saying the default ns is called 'default'? Or that it has no name?

It has no name similar to the java default package.

Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)

I wish too. We can refactor after this to create one?

How much of this design has to be in place for 0.96 to ship?

We're watching out for backwards compatibility? I believe core namespace should be enough. We can put the rest in 96.1, quota stuff needs addition to the cli. Security has some changes as well to support namespaces but it's just an extension.

Below is interesting (smile)

It's our secret version

Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).

My line of thinking was that namespace is a component of tableName ie "tableName = <table namespace>.<table qualifier>". Just like column = CF:CQ.

NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

Where did you see that? Group information is added as a KV pair to the NamespaceDesc.

Francis Liu
added a comment - 09/May/13 00:04
This can be done later, right? You'd have to do it in hbase.
Yep
Are we saying the default ns is called 'default'? Or that it has no name?
It has no name similar to the java default package.
Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)
I wish too. We can refactor after this to create one?
How much of this design has to be in place for 0.96 to ship?
We're watching out for backwards compatibility? I believe core namespace should be enough. We can put the rest in 96.1, quota stuff needs addition to the cli. Security has some changes as well to support namespaces but it's just an extension.
Below is interesting (smile)
It's our secret version
Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).
My line of thinking was that namespace is a component of tableName ie "tableName = <table namespace>.<table qualifier>". Just like column = CF:CQ.
NamespaceDesc.. has notion of groups. Is that right given they are add on for later?
Where did you see that? Group information is added as a KV pair to the NamespaceDesc.
NamespaceDescriptor implements java Serializable
Oops will remove that

Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)

Enis Soztutar do you have any bandwidth? Since you've shown interested earlier

IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.

Looking at the comments there aren't any major issues apart from the delimiter. The namespace affected tests are passing.

Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

Either sounds fine to me.

Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)

Working on my existing patch we can try out elliot's suggestion which would essentially require the new interfaces

If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.

Yep it's possible. Though the patch will need some small changes in assignment manager and LoadBalancer interface, etc. Better to get those in soon if we're in a hurry. I can separate those out as a separate patch as well?

Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).

Francis Liu
added a comment - 08/May/13 23:51
Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)
Enis Soztutar do you have any bandwidth? Since you've shown interested earlier
IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.
Looking at the comments there aren't any major issues apart from the delimiter. The namespace affected tests are passing.
Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.
Either sounds fine to me.
Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)
Working on my existing patch we can try out elliot's suggestion which would essentially require the new interfaces
If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.
Yep it's possible. Though the patch will need some small changes in assignment manager and LoadBalancer interface, etc. Better to get those in soon if we're in a hurry. I can separate those out as a separate patch as well?
Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).
It's gonna be hard, will give this a try.

Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)

IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.

Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)

If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.

+1 on getting rid of funky system tables stuff (but yeah, as per Francis above, they were given the funny names so they sorted before user tables).

Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).

On the design, needs date, author, and JIRA number.

As a first step we only intend to limit the number of tables and regions a given namespace may contain.

This can be done later, right? You'd have to do it in hbase.

Are we saying the default ns is called 'default'? Or that it has no name?

stack
added a comment - 08/May/13 23:09 Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)
IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.
Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.
Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)
If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.
+1 on getting rid of funky system tables stuff (but yeah, as per Francis above, they were given the funny names so they sorted before user tables).
Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).
On the design, needs date, author, and JIRA number.
As a first step we only intend to limit the number of tables and regions a given namespace may contain.
This can be done later, right? You'd have to do it in hbase.
Are we saying the default ns is called 'default'? Or that it has no name?
NamespaceController coprocessor running on region servers to correctly enforce quota restrictions
Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)
How much of this design has to be in place for 0.96 to ship?
<ROOT>/data/<NAMESPACE>/<TABLE>
Why the need for 'data' in above?
Looking quickly at the attached patch (will look at rb later):
Below is interesting (smile)
- <hadoop.version>0.23.3</hadoop.version>
+ <hadoop.version>0.23.7-SNAPSHOT</hadoop.version>
Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).
NamespaceDescriptor implements java Serializable?
NamespaceDesc.. has notion of groups. Is that right given they are add on for later?
Will look at rb patch next. What is attached doesn't seem enough. Will be back.

Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

Essentially we need to keep the "HTable(byte[] tableName, Configuration config)" constructor. The current implementation would assume a table is part of the default namespace if only the table name is provided.

Apart from potentially confusing our users. Do you have any other concerns against supporting a fully-qualified table name?

Francis Liu
added a comment - 08/May/13 04:31
Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).
Essentially we need to keep the "HTable(byte[] tableName, Configuration config)" constructor. The current implementation would assume a table is part of the default namespace if only the table name is provided.
Apart from potentially confusing our users. Do you have any other concerns against supporting a fully-qualified table name?

Should we support cross-namespace constructs ? If so, namespace would be exposed to user.

From my point of view the most important usability concern should be the java api, with thrift second. My proposal would allow cross namespace very easily by just creating two different HTable's in different namespaces.

The shell is nice but it's just a convenience utility. We shouldn't make the most used case harder or more confusing chasing some dream of having a sql shell. In my opinion since HBase is not Oracle's 11g, it shouldn't value being like their sql.

We need to expose it mainly for backwards compatibility.

Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

Elliott Clark
added a comment - 08/May/13 04:16 Should we support cross-namespace constructs ? If so, namespace would be exposed to user.
From my point of view the most important usability concern should be the java api, with thrift second. My proposal would allow cross namespace very easily by just creating two different HTable's in different namespaces.
The shell is nice but it's just a convenience utility. We shouldn't make the most used case harder or more confusing chasing some dream of having a sql shell. In my opinion since HBase is not Oracle's 11g, it shouldn't value being like their sql.
We need to expose it mainly for backwards compatibility.
Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

I see. We need to expose it mainly for backwards compatibility. Also isn't being able to reference a fully-qualified table name common practice in databases? So users aren't learning something new here.

Francis Liu
added a comment - 08/May/13 03:40 I see. We need to expose it mainly for backwards compatibility. Also isn't being able to reference a fully-qualified table name common practice in databases? So users aren't learning something new here.

Ted Yu
added a comment - 08/May/13 03:40 Should we support cross-namespace constructs ? If so, namespace would be exposed to user.
For Oracle, some Namespaces are reserved:
http://docs.oracle.com/cd/B28359_01/appdev.111/b31231/appb.htm#BABFFDFC
This means an ordinary user query can easily reference more than one namespace.

At the very least we will have to store table names fully qualified in the catalog tables.

True, but we don't need to use . here, or expose it to the user.
So I'm saying most places should use a different character to store the table name, and I would argue that we shouldn't expose that fully qualified table name to the user at all.

With a namespace

create 'namespace', 't1', 'f1'

Places something in meta like: 'namespace,t1,,hash'

Put Usage:

put 'namespace', 't1', 'r1', 'c1', 'value'

or

use 'namespace'
put 't1', 'r1', 'c1', 'value'

default space

create 't1', 'f1'

places something in meta like: 'default,t1,,hash'

put 't1', 'r1', 'c1', 'value'

or

put 'default', 't1', 'r1', 'c1', 'value'

Comma is already reserved and we can use it for everywhere that we need a string name. But we shouldn't expose that to the user. We shouldn't overload table name with two different parts. It gets confusing for users when one parameter actually represents two different things (namespace and table name).

Elliott Clark
added a comment - 08/May/13 02:45 - edited At the very least we will have to store table names fully qualified in the catalog tables.
True, but we don't need to use . here, or expose it to the user.
So I'm saying most places should use a different character to store the table name, and I would argue that we shouldn't expose that fully qualified table name to the user at all.
With a namespace
create 'namespace', 't1', 'f1'
Places something in meta like: 'namespace,t1,,hash'
Put Usage:
put 'namespace', 't1', 'r1', 'c1', 'value'
or
use 'namespace'
put 't1', 'r1', 'c1', 'value'
default space
create 't1', 'f1'
places something in meta like: 'default,t1,,hash'
put 't1', 'r1', 'c1', 'value'
or
put ' default ', 't1', 'r1', 'c1', 'value'
Comma is already reserved and we can use it for everywhere that we need a string name. But we shouldn't expose that to the user. We shouldn't overload table name with two different parts. It gets confusing for users when one parameter actually represents two different things (namespace and table name).

Elliott Clark We can add that an the interface level to support both means of identifying tables. At the very least we will have to store table names fully qualified in the catalog tables. Storing it on hdfs and zk the same way makes things easier as well.

By default namespace did you mean something like a "use <namespace>" command in the cli?

Francis Liu
added a comment - 08/May/13 02:20 Elliott Clark We can add that an the interface level to support both means of identifying tables. At the very least we will have to store table names fully qualified in the catalog tables. Storing it on hdfs and zk the same way makes things easier as well.
By default namespace did you mean something like a "use <namespace>" command in the cli?

I think that adding a parameter for namespace is cleaner. It would mean that we don't add need to rename tables. People can continue to keep their table names. We also then get an easier path to infer the default namespace.

Elliott Clark
added a comment - 08/May/13 01:10 I think that adding a parameter for namespace is cleaner. It would mean that we don't add need to rename tables. People can continue to keep their table names. We also then get an easier path to infer the default namespace.

Another option is to factor the helper piece out of this class and get that committed quicker. Apart from extra work, we won't be able to verify everything is covered without the rest of the patch. Thoughts?

Francis Liu
added a comment - 08/May/13 00:42 Another option is to factor the helper piece out of this class and get that committed quicker. Apart from extra work, we won't be able to verify everything is covered without the rest of the patch. Thoughts?

Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.

stack Would we be able to get it in quicker with a feature branch? The reason I am asking is because part of what's challenging about this enhancement is catching all the code which makes assumption on path changes. And that will continue to grow until we provide the helper functions to be used. We have an 0.94 version of this patch working in sandbox with active users. So I believe basic functionality is working. The big thing missing is the migration script (We don't use '.' in table names).

Francis Liu
added a comment - 08/May/13 00:24
Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.
stack Would we be able to get it in quicker with a feature branch? The reason I am asking is because part of what's challenging about this enhancement is catching all the code which makes assumption on path changes. And that will continue to grow until we provide the helper functions to be used. We have an 0.94 version of this patch working in sandbox with active users. So I believe basic functionality is working. The big thing missing is the migration script (We don't use '.' in table names).

Ted Yu
added a comment - 07/May/13 22:38 @Francis:
Take a look at the summary here:
http://search-hadoop.com/m/K0wzj1V5vpu
Delimiter and directory structure for namespace support is discussed in the middle of the above summary.

For security, should each namespace have its own permission settings ?

Yes, the permissions are similar to to directory privileges. You would need write access to a namespace to be able to create and drop tables. We already have a patch for this in 0.94. We'll put it up once we sort out the core patch.

For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?

yep see subtask. We also have this implemented will put up patch once most things are sorted out with core patch.

Francis Liu
added a comment - 07/May/13 22:38
For security, should each namespace have its own permission settings ?
Yes, the permissions are similar to to directory privileges. You would need write access to a namespace to be able to create and drop tables. We already have a patch for this in 0.94. We'll put it up once we sort out the core patch.
For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?
yep see subtask. We also have this implemented will put up patch once most things are sorted out with core patch.

@Francis:
Can you add new tests for this feature ?
The new tests should include unit tests and, integration tests.

There are unit tests I can have it run as an IT test as well. Though the real test really is the existing tests as the main issues are updating/replacing code segments which make assumptions about the directory layout. The tricky ones were in snapshots, hbck and splits. In this patch I've gotten all of those to pass. We can add more tests to these components as needed.

Francis Liu
added a comment - 07/May/13 22:36
@Francis:
Can you add new tests for this feature ?
The new tests should include unit tests and, integration tests.
There are unit tests I can have it run as an IT test as well. Though the real test really is the existing tests as the main issues are updating/replacing code segments which make assumptions about the directory layout. The tricky ones were in snapshots, hbck and splits. In this patch I've gotten all of those to pass. We can add more tests to these components as needed.

Francis Liu
added a comment - 07/May/13 22:31
Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ?
When I was asking about a different delimiter. No one was really happy with anything else but a ".". If you have a suggestion please let us know.

Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.

It has been suggested that we make a branch to speed dev on this issue which should help move things along; a branch will need a sponsor/conmiitter. I could volunteer but there are probably better mentors than sloppy I who can help along this issue and advise on what is needed to make the cut-off date fast looming.

stack
added a comment - 05/May/13 06:51 Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.
It has been suggested that we make a branch to speed dev on this issue which should help move things along; a branch will need a sponsor/conmiitter. I could volunteer but there are probably better mentors than sloppy I who can help along this issue and advise on what is needed to make the cut-off date fast looming.

Ted Yu
added a comment - 03/May/13 23:38 Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ?
For security, should each namespace have its own permission settings ?
For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?

Francis Liu
added a comment - 30/Apr/13 01:55
Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ?
That's disk space quota. Right now we plan to enforce it using hdfs quota and see how that goes. That way we don't have to reinvent the wheel.
What about naming the new command create_namespace (with verb at the beginning) ?
I wanted to namespace the commands . Let me know if the convention is different for HBase.

Ted Yu
added a comment - 30/Apr/13 00:16 Going over the design doc, I have a few questions:
As a first step we only intend to limit the number of tables and regions a given namespace may contain.
Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ?
namespace_create ‘<namespace>’, {
What about naming the new command create_namespace (with verb at the beginning) ?

Enis Soztutar I would very much like to get it in as well. I've addressed your comments. Also I've broken down the pieces into subtasks. The patch on RB is for the first subtask. We currently have 1-3 running on 0.94. We'll forward port the 2nd and 3rd piece as soon as the first task completed to keep things manageable.

Francis Liu
added a comment - 24/Apr/13 00:24 Enis Soztutar I would very much like to get it in as well. I've addressed your comments. Also I've broken down the pieces into subtasks. The patch on RB is for the first subtask. We currently have 1-3 running on 0.94. We'll forward port the 2nd and 3rd piece as soon as the first task completed to keep things manageable.

Francis Liu, I would like to get namespaces in 0.96, because it fits well into the singularity. Getting it in later would be hard. I've put up some comments at RB. I can offer some help should you need with review / split / etc.

Enis Soztutar
added a comment - 11/Apr/13 23:14 Francis Liu , I would like to get namespaces in 0.96, because it fits well into the singularity. Getting it in later would be hard. I've put up some comments at RB. I can offer some help should you need with review / split / etc.

Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

AFAIK, if the META table goes offline due to RS failure. There is no special case to make it's assignment a priority. And priority is based on lexical ordering, in which case we will need the funky prefix. Not unless we'd want to add complexity to handle that logic.

BTW the check is not in yet. But I'd also like to remove '.' as an option for table name. Beyond the proposed namespace delimiter usage.

Francis Liu
added a comment - 29/Mar/13 00:10
Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.
AFAIK, if the META table goes offline due to RS failure. There is no special case to make it's assignment a priority. And priority is based on lexical ordering, in which case we will need the funky prefix. Not unless we'd want to add complexity to handle that logic.
BTW the check is not in yet. But I'd also like to remove '.' as an option for table name. Beyond the proposed namespace delimiter usage.

Francis Liu
added a comment - 28/Mar/13 23:54
TestMasterFailover failed with patch. In test output, I saw:
Yep, as I mentioned in reviewboard. There are 3 tests that are failing. Have yet to determine if it is related to this patch.

Francis Liu
added a comment - 28/Mar/13 23:53
I am still going through your code.
Would user be able to create table in default namespace after migration ?
Put in another way: what privilege is required to do the above ?
namespace affiliation is determine by table name during creation.
ie
#create table in namespace foo
create 'foo.my_table','f'
#create table in namespace default
create 'my_table','f'

Ted Yu
added a comment - 28/Mar/13 22:39 It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed.
I am still going through your code.
Would user be able to create table in default namespace after migration ?
Put in another way: what privilege is required to do the above ?

Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

Enis Soztutar
added a comment - 28/Mar/13 22:08 I've renamed the meta table to become -hbase.meta
Nice. This is the same as what was proposed over in HBASE-8093
Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.

It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed. They will all fall into the default namespace bucket and be accessed by the application the same way.

Francis Liu
added a comment - 28/Mar/13 21:29 Thanks for reviewing ted.
What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.
It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed. They will all fall into the default namespace bucket and be accessed by the application the same way.

Ted Yu
added a comment - 28/Mar/13 20:49 I am beginning to go over the code on review board.
-there's a special case with tables in the default namespace: <table qualifier>
What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.
I've renamed the meta table to become -hbase . meta
Nice. This is the same as what was proposed over in HBASE-8093

Francis Liu
added a comment - 28/Mar/13 17:24
Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group).
I find the latter more intuitive.
If you find that agreeable. We can keep it that way.
Would the group table from HBASE­6721 belong to the above namespace ?
Yep. depending on which ends up going in first will have to make that change.

Sorry for the delay I was OOO last week. The patch is way bigger than I anticipated changing the path hierarchy took bulk of it as assumptions about it was all over the code. Here's an initial draft of the patch, functionally it's complete. I need some feedback before I can go further.

Francis Liu
added a comment - 28/Mar/13 17:19 Sorry for the delay I was OOO last week. The patch is way bigger than I anticipated changing the path hierarchy took bulk of it as assumptions about it was all over the code. Here's an initial draft of the patch, functionally it's complete. I need some feedback before I can go further.
https://reviews.apache.org/r/10167/diff/#index_header

Ted Yu
added a comment - 08/Mar/13 01:01 Some questions on design doc.
multiple namespaces can be part of the same region server group.
Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group).
I find the latter more intuitive.
hbase ­ admin tables:­ROOT­, .META., ACLs
Would the group table from HBASE­6721 belong to the above namespace ?

How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible.

Maybe we should have WALs per namespace per server . If we want other namespace based features (ie replication) going this route makes sense?

Internally we plan to do it using regionserver groups (one namespace per group). Since we don't want our tenants contending for other resources as well (disk, mem, cpu, etc).

Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

It'd be good if we can leverage hdfs quota since it's already there and has worked well (at least for us). That's also less complexity for us, then we can focus on adding HBase specific Quota (ie regions, tables, etc).

Francis Liu
added a comment - 07/Mar/13 22:31
How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible.
Maybe we should have WALs per namespace per server . If we want other namespace based features (ie replication) going this route makes sense?
Internally we plan to do it using regionserver groups (one namespace per group). Since we don't want our tenants contending for other resources as well (disk, mem, cpu, etc).
Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.
It'd be good if we can leverage hdfs quota since it's already there and has worked well (at least for us). That's also less complexity for us, then we can focus on adding HBase specific Quota (ie regions, tables, etc).

I've also changed the tableDir on hdfs to include namespace as the parent directory

How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible. Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

Enis Soztutar
added a comment - 07/Mar/13 19:31 I've also changed the tableDir on hdfs to include namespace as the parent directory
How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible. Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

So it seems we're fine providing a migration script. So I've gone ahead and produced a rough patch with the change.

The general idea is to have a TableName class which encapsulates both namespace and tableQualifier (AKA the old table name). And this will be what gets passed around as much as possible internally. Table names are referenced fully qualified otherwise.
I've also changed the tableDir on hdfs to include namespace as the parent directory. This will enable us to set HDFS quotas by namespace.

I skipped the CRUD related stuff since that seems pretty straightforward. Let me know if this approach is acceptable.

Francis Liu
added a comment - 07/Mar/13 18:55 So it seems we're fine providing a migration script. So I've gone ahead and produced a rough patch with the change.
The general idea is to have a TableName class which encapsulates both namespace and tableQualifier (AKA the old table name). And this will be what gets passed around as much as possible internally. Table names are referenced fully qualified otherwise.
I've also changed the tableDir on hdfs to include namespace as the parent directory. This will enable us to set HDFS quotas by namespace.
I skipped the CRUD related stuff since that seems pretty straightforward. Let me know if this approach is acceptable.

Jesse Yates
added a comment - 07/Mar/13 02:13 Francis Liu what kind of timeline are you planning for doing this work? I love to see it subsume HBASE-7999 , but this is a much more involved change than what's going on there.

Jesse Yates
added a comment - 07/Mar/13 02:09 - edited my point about backward compat was about the storage
IIRC we don't support backward compat for 0.96 backwards. As long as this goes into trunk, it should be fine going forward.

Francis Liu
added a comment - 06/Mar/13 22:42
The latter may not necessarily be backward compatible...
Can you give an example to this? Thinking about it more, it seems to me you'll just end up with a lot of namespaces?

I see, I am assuming we have to store namespace as part of the region name. And store them fully-qualified on hdfs/zookeeper/etc. Else we would be forced to have all table names to be globally unique which would different from database semantics.

Another concern is if a user can run the same application code against 0.96.

ie if I wanted to scan:

> scan 'foo.bar'

Pre-NS this would scan table 'foo.bar'. Post-NS, the system would parse this out as table bar in namespace foo. One way we could deal with this is read the table as Post-NS if it doesn't access read it as Pre-NS, check again if not then fail.

Francis Liu
added a comment - 06/Mar/13 22:39 I see, I am assuming we have to store namespace as part of the region name. And store them fully-qualified on hdfs/zookeeper/etc. Else we would be forced to have all table names to be globally unique which would different from database semantics.
Another concern is if a user can run the same application code against 0.96.
ie if I wanted to scan:
> scan 'foo.bar'
Pre-NS this would scan table 'foo.bar'. Post-NS, the system would parse this out as table bar in namespace foo. One way we could deal with this is read the table as Post-NS if it doesn't access read it as Pre-NS, check again if not then fail.

my point about backward compat was about the storage, not display. I.e. if people want to create a table "a.b" without namespace and be confused while viewing some table list, I think it's ok. Maybe we can disallow tables with dots.
If they have existing table "a.b" and NS is stored separately, again the worst thing that happens is user gets confused, reads the release notes and renames it. Should be ok to.
But if we /store/ NS in table name, then with table a.b, without special handing, system gets confused (thinks it's in NS "a") after upgrade

Sergey Shelukhin
added a comment - 06/Mar/13 21:48 my point about backward compat was about the storage, not display. I.e. if people want to create a table "a.b" without namespace and be confused while viewing some table list, I think it's ok. Maybe we can disallow tables with dots.
If they have existing table "a.b" and NS is stored separately, again the worst thing that happens is user gets confused, reads the release notes and renames it. Should be ok to.
But if we /store/ NS in table name, then with table a.b, without special handing, system gets confused (thinks it's in NS "a") after upgrade

Francis Liu
added a comment - 06/Mar/13 19:25 Given that '.' is already used. Should we pick another delimiter for namespaces? Or should we provide a backward compatible way to support this. Ie creating namespaces for tablenames with '.'?

I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

They will be first class citizens. There will be a namespace table for namespace meta information. Also embedding namespace information in table name does not prevent this, we are just introducing the notion of fully-qualified table names (which we should introduce anyway). And store them in this form in meta/root.

Francis Liu
added a comment - 06/Mar/13 19:23
I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.
They will be first class citizens. There will be a namespace table for namespace meta information. Also embedding namespace information in table name does not prevent this, we are just introducing the notion of fully-qualified table names (which we should introduce anyway). And store them in this form in meta/root.

Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

Enis Soztutar
added a comment - 06/Mar/13 18:57 Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this
I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

Thanks for the clarification. It seems like '.' and '-' isn't allowed only if it's the first character. For backward compatibility why don't we create namespaces for those tables that are named that way?

Francis Liu
added a comment - 06/Mar/13 18:46
w.r.t. dot in table name:
public static final String VALID_USER_TABLE_REGEX = "(?: [a-zA-Z_0-9] [a-zA-Z_0-9.-] *)";
So dot should be allowed
Thanks for the clarification. It seems like '.' and '-' isn't allowed only if it's the first character. For backward compatibility why don't we create namespaces for those tables that are named that way?

Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.

"Given the level of autonomy namespaces will provide tenants. " what does this mean?

In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.

Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.

Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?

Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?

Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

Francis Liu
added a comment - 06/Mar/13 18:38 reposting coz of typos:
+1 on making core (as opposed to cp-s I assume?)
Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.
"Given the level of autonomy namespaces will provide tenants. " what does this mean?
In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.
Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.
Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this
Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?
Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?
Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?
As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

+1 on having namespaces in core. Namespace/database's are universally understood in terms of the database space. We can keep the grouping of regionservers per namespace out of core, and deliver that as a part of the other region grouping issue.

Enis Soztutar
added a comment - 06/Mar/13 18:35 +1 on having namespaces in core. Namespace/database's are universally understood in terms of the database space. We can keep the grouping of regionservers per namespace out of core, and deliver that as a part of the other region grouping issue.

Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.

"Given the level of autonomy namespaces will provide tenants. " what does this mean?

In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.

{quota}
Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.{quota}

Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

{quota}
Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?{quota}

Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?

{quota}
Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?{quota}

As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

Francis Liu
added a comment - 06/Mar/13 18:34
+1 on making core (as opposed to cp-s I assume?)
Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.
"Given the level of autonomy namespaces will provide tenants. " what does this mean?
In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.
{quota}
Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.{quota}
Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this
{quota}
Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?{quota}
Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?
{quota}
Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?{quota}
As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

+1 on making core (as opposed to cp-s I assume?)
"Given the level of autonomy namespaces will provide tenants. " what does this mean?
What kind of resource allocation will be possible, except for server groups (just examples, to understand it better)?

Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.
Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?
Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

Sergey Shelukhin
added a comment - 06/Mar/13 18:12 +1 on making core (as opposed to cp-s I assume?)
"Given the level of autonomy namespaces will provide tenants. " what does this mean?
What kind of resource allocation will be possible, except for server groups (just examples, to understand it better)?
Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.
Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?
Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

Francis Liu
added a comment - 06/Mar/13 17:55 Initial draft of design. This originally was intended to be implemented as coprocessors thus it's design was made to be as non-invasive as possible.
Enis Soztutar Suggested that it would be better to make this part of core. I'd be up for doing that and open to other changes to make things more integrated.