I am of the opinion this feature is too specialized to be generally helpful.

-------------------------------The cardinality of 'x' is in 1000's per partition of T. Moreover,there is a skew for the values of 'x'. In general, there are ~10values of 'x' which have a very large skew, and the remainingvalues of 'x' have a small cardinality. Also, note that this mapping(values of 'x' with a high cardinality can change daily).--------------------------

In these cases you should use clustering/bucketing. This will preventthe skew you are talking about. If you want more efficiency in certainquery types build a index on top of the original table.

I understand someone wanting to do this because mysql partition can dothis, but this sounds like a management problem. Who is to say theskew is the same each partition?

-----------------------------------------hive compiler to do input pruning. The list of skewed keys is storedat the table level (note that, this list can be initially supplied bythe client periodically, and can be eventually updated when a newpartition is being loaded).-----------------------------------------

Imagine you have a table partitioned by hour and two datacenters Chinaand NY. At some hours the skew will be different. Skews change overtime. Since this property is table level I do not understand how thiswould be changed.

This feature is designed for use cases described in wiki. We do see themin the real life so that we come up with the feature.

In this first release, in order to use the feature:1. Hive table users need to know the skewed key in advance2. Hive table users need to know the skewed key is the same eachpartition. 3. If Hive table users know skewed key change, they can "alter" skewed keyvia "alter" statement.

4. If #3 happens, old partitions have old skewed key and new partitionhave new. It's expected.

We may consider the following in the future release:1. Hive instruments skewed key and displays them to user

Thanks

TimOn 6/14/12 9:34 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:

>I am of the opinion this feature is too specialized to be generally>helpful.>>------------------------------->The cardinality of 'x' is in 1000's per partition of T. Moreover,>there is a skew for the values of 'x'. In general, there are ~10>values of 'x' which have a very large skew, and the remaining>values of 'x' have a small cardinality. Also, note that this mapping>(values of 'x' with a high cardinality can change daily).>-------------------------->>In these cases you should use clustering/bucketing. This will prevent>the skew you are talking about. If you want more efficiency in certain>query types build a index on top of the original table.>>I understand someone wanting to do this because mysql partition can do>this, but this sounds like a management problem. Who is to say the>skew is the same each partition?>>----------------------------------------->hive compiler to do input pruning. The list of skewed keys is stored>at the table level (note that, this list can be initially supplied by>the client periodically, and can be eventually updated when a new>partition is being loaded).>----------------------------------------->>Imagine you have a table partitioned by hour and two datacenters China>and NY. At some hours the skew will be different. Skews change over>time. Since this property is table level I do not understand how this>would be changed.>>>>On Thu, Jun 14, 2012 at 4:14 AM, Carl Steinbach <[EMAIL PROTECTED]> wrote:>> Hi Tim,>>>> I added some comments to the wiki a couple days ago. I just wanted to>>make>> sure you saw them since it doesn't look like you're registered as a>>watcher>> for that page.>>>> Thanks.>>>> Carl>>>> On Mon, Jun 11, 2012 at 12:22 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>>>>> Hi Carl, thanks Tim>>>>>> On 6/11/12 12:14 PM, "Carl Steinbach" <[EMAIL PROTECTED]> wrote:>>>>>> >+ hcatalog-dev>>> >>>> >On Mon, Jun 11, 2012 at 12:09 PM, Carl Steinbach <[EMAIL PROTECTED]>>>> >wrote:>>> >>>> >> This link may work better for some people:>>> >>>>> >> https://cwiki.apache.org/confluence/display/Hive/ListBucketing>>> >>>>> >> Thanks.>>> >>>>> >> Carl>>> >>>>> >>>>> >> On Mon, Jun 11, 2012 at 12:03 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>>> >>>>> >>> Dear all hive developers,>>> >>>>>> >>> We are making good progress of implementing the list bucketing>>> >>>feature. It>>> >>> should be available soon in weeks.>>> >>>>>> >>> We'd like to call feature review again and please provide your>>> >>>comments.>>> >>>>>> >>> Thanks>>> >>>>>> >>> Tim>>> >>>>>> >>> On 6/1/12 10:13 AM, "Gang Liu" <[EMAIL PROTECTED]> wrote:>>> >>>>>> >>> >Dear all,>>> >>> >>>> >>> >Please review the proposal and provide your comments:>>> >>> >>>> >>> >https://cwiki.apache.org/Hive/listbucketing.html>>> >>> >>>> >>> >>>> >>> >Thanks>>> >>> >>>> >>> >Tim>>> >>> >>>> >>>>>> >>>>>> >>>>>>>>

We have had a ticket open for quite some time for combine input formatto work across partitions. Not sure if that can help with what you areseeing as well. It could help us alot.

Edward

On Thu, Jun 14, 2012 at 1:25 PM, Gang Liu <[EMAIL PROTECTED]> wrote:> Hey Edward,>> Thank you very much for providing comments.>> This feature is designed for use cases described in wiki. We do see them> in the real life so that we come up with the feature.>> In this first release, in order to use the feature:> 1. Hive table users need to know the skewed key in advance> 2. Hive table users need to know the skewed key is the same each> partition.> 3. If Hive table users know skewed key change, they can "alter" skewed key> via "alter" statement.>> 4. If #3 happens, old partitions have old skewed key and new partition> have new. It's expected.>> We may consider the following in the future release:> 1. Hive instruments skewed key and displays them to user>> Thanks>> Tim>>> On 6/14/12 9:34 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:>>>I am of the opinion this feature is too specialized to be generally>>helpful.>>>>------------------------------->>The cardinality of 'x' is in 1000's per partition of T. Moreover,>>there is a skew for the values of 'x'. In general, there are ~10>>values of 'x' which have a very large skew, and the remaining>>values of 'x' have a small cardinality. Also, note that this mapping>>(values of 'x' with a high cardinality can change daily).>>-------------------------->>>>In these cases you should use clustering/bucketing. This will prevent>>the skew you are talking about. If you want more efficiency in certain>>query types build a index on top of the original table.>>>>I understand someone wanting to do this because mysql partition can do>>this, but this sounds like a management problem. Who is to say the>>skew is the same each partition?>>>>----------------------------------------->>hive compiler to do input pruning. The list of skewed keys is stored>>at the table level (note that, this list can be initially supplied by>>the client periodically, and can be eventually updated when a new>>partition is being loaded).>>----------------------------------------->>>>Imagine you have a table partitioned by hour and two datacenters China>>and NY. At some hours the skew will be different. Skews change over>>time. Since this property is table level I do not understand how this>>would be changed.>>>>>>>>On Thu, Jun 14, 2012 at 4:14 AM, Carl Steinbach <[EMAIL PROTECTED]> wrote:>>> Hi Tim,>>>>>> I added some comments to the wiki a couple days ago. I just wanted to>>>make>>> sure you saw them since it doesn't look like you're registered as a>>>watcher>>> for that page.>>>>>> Thanks.>>>>>> Carl>>>>>> On Mon, Jun 11, 2012 at 12:22 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>>>>>>> Hi Carl, thanks Tim>>>>>>>> On 6/11/12 12:14 PM, "Carl Steinbach" <[EMAIL PROTECTED]> wrote:>>>>>>>> >+ hcatalog-dev>>>> >>>>> >On Mon, Jun 11, 2012 at 12:09 PM, Carl Steinbach <[EMAIL PROTECTED]>>>>> >wrote:>>>> >>>>> >> This link may work better for some people:>>>> >>>>>> >> https://cwiki.apache.org/confluence/display/Hive/ListBucketing>>>> >>>>>> >> Thanks.>>>> >>>>>> >> Carl>>>> >>>>>> >>>>>> >> On Mon, Jun 11, 2012 at 12:03 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>>>> >>>>>> >>> Dear all hive developers,>>>> >>>>>>> >>> We are making good progress of implementing the list bucketing>>>> >>>feature. It>>>> >>> should be available soon in weeks.>>>> >>>>>>> >>> We'd like to call feature review again and please provide your>>>> >>>comments.>>>> >>>>>>> >>> Thanks>>>> >>>>>>> >>> Tim>>>> >>>>>>> >>> On 6/1/12 10:13 AM, "Gang Liu" <[EMAIL PROTECTED]> wrote:>>>> >>>>>>> >>> >Dear all,>>>> >>> >>>>> >>> >Please review the proposal and provide your comments:>>>> >>> >>>>> >>> >https://cwiki.apache.org/Hive/listbucketing.html>>>> >>> >>>>> >>> >>>>> >>> >Thanks>>>> >>

>We have had a ticket open for quite some time for combine input format>to work across partitions. Not sure if that can help with what you are>seeing as well. It could help us alot.>>Edward>>On Thu, Jun 14, 2012 at 1:25 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>> Hey Edward,>>>> Thank you very much for providing comments.>>>> This feature is designed for use cases described in wiki. We do see them>> in the real life so that we come up with the feature.>>>> In this first release, in order to use the feature:>> 1. Hive table users need to know the skewed key in advance>> 2. Hive table users need to know the skewed key is the same each>> partition.>> 3. If Hive table users know skewed key change, they can "alter" skewed>>key>> via "alter" statement.>>>> 4. If #3 happens, old partitions have old skewed key and new partition>> have new. It's expected.>>>> We may consider the following in the future release:>> 1. Hive instruments skewed key and displays them to user>>>> Thanks>>>> Tim>>>>>> On 6/14/12 9:34 AM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:>>>>>I am of the opinion this feature is too specialized to be generally>>>helpful.>>>>>>------------------------------->>>The cardinality of 'x' is in 1000's per partition of T. Moreover,>>>there is a skew for the values of 'x'. In general, there are ~10>>>values of 'x' which have a very large skew, and the remaining>>>values of 'x' have a small cardinality. Also, note that this mapping>>>(values of 'x' with a high cardinality can change daily).>>>-------------------------->>>>>>In these cases you should use clustering/bucketing. This will prevent>>>the skew you are talking about. If you want more efficiency in certain>>>query types build a index on top of the original table.>>>>>>I understand someone wanting to do this because mysql partition can do>>>this, but this sounds like a management problem. Who is to say the>>>skew is the same each partition?>>>>>>----------------------------------------->>>hive compiler to do input pruning. The list of skewed keys is stored>>>at the table level (note that, this list can be initially supplied by>>>the client periodically, and can be eventually updated when a new>>>partition is being loaded).>>>----------------------------------------->>>>>>Imagine you have a table partitioned by hour and two datacenters China>>>and NY. At some hours the skew will be different. Skews change over>>>time. Since this property is table level I do not understand how this>>>would be changed.>>>>>>>>>>>>On Thu, Jun 14, 2012 at 4:14 AM, Carl Steinbach <[EMAIL PROTECTED]>>>>wrote:>>>> Hi Tim,>>>>>>>> I added some comments to the wiki a couple days ago. I just wanted to>>>>make>>>> sure you saw them since it doesn't look like you're registered as a>>>>watcher>>>> for that page.>>>>>>>> Thanks.>>>>>>>> Carl>>>>>>>> On Mon, Jun 11, 2012 at 12:22 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>>>>>>>>> Hi Carl, thanks Tim>>>>>>>>>> On 6/11/12 12:14 PM, "Carl Steinbach" <[EMAIL PROTECTED]> wrote:>>>>>>>>>> >+ hcatalog-dev>>>>> >>>>>> >On Mon, Jun 11, 2012 at 12:09 PM, Carl Steinbach <[EMAIL PROTECTED]>>>>>> >wrote:>>>>> >>>>>> >> This link may work better for some people:>>>>> >>>>>>> >> https://cwiki.apache.org/confluence/display/Hive/ListBucketing>>>>> >>>>>>> >> Thanks.>>>>> >>>>>>> >> Carl>>>>> >>>>>>> >>>>>>> >> On Mon, Jun 11, 2012 at 12:03 PM, Gang Liu <[EMAIL PROTECTED]> wrote:>>>>> >>>>>>> >>> Dear all hive developers,>>>>> >>>>>>>> >>> We are making good progress of implementing the list bucketing>>>>> >>>feature. It>>>>> >>> should be available soon in weeks.>>>>> >>>>>>>> >>> We'd like to call feature review again and please provide your>>>>> >>>comments.>>>>> >>>>>>>> >>> Thanks>>>>> >>>>>>>> >>> Tim>>>>> >>>>>>>> >>> On 6/1/12 10:13 AM, "Gang Liu" <[EMAIL PROTECTED]> wrote:

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext