Quick clarification on TTL for cells. The concept makes sense (instead of "keep 3 versions" you say "keep versions more recent than time T"). But, if there's only 1 value in the cell, and that value is older than the TTL, will it also be deleted?

If so, has there ever been discussion of a "TTL except for most recent" option? (i.e. you want the current version to be permanently persistent, but also want some time-based range of version history, so you can peek back and get consistent snapshots within the last hour, 6 hours, 24 hours, etc). TTL seems perfect for this, but not if it'll chomp the current version of cells too! :)

Quick clarification on TTL for cells. The concept makes sense (instead of "keep 3 versions" you say "keep versions more recent than time T"). But, if there's only 1 value in the cell, and that value is older than the TTL, will it also be deleted?

If so, has there ever been discussion of a "TTL except for most recent" option? (i.e. you want the current version to be permanently persistent, but also want some time-based range of version history, so you can peek back and get consistent snapshots within the last hour, 6 hours, 24 hours, etc). TTL seems perfect for this, but not if it'll chomp the current version of cells too! :)

I like the idea of making this pluggable (via the coprocessor framework, or otherwise). But I also think this is a fundamental enough policy option that making it hard-coded might be a good idea. When I was talking to someone the other day about the current TTL policy, he was like, "WTF, who would want that, it eats your data?". There's no such thing as a "keep 0 versions" option, and thus no way to accidentally lose your most current data using that approach. But with the TTL version there is, which is (IMO) counter-intuitive for those coming from an RDBMS background.

Quick clarification on TTL for cells. The concept makes sense (instead of "keep 3 versions" you say "keep versions more recent than time T"). But, if there's only 1 value in the cell, and that value is older than the TTL, will it also be deleted?

If so, has there ever been discussion of a "TTL except for most recent" option? (i.e. you want the current version to be permanently persistent, but also want some time-based range of version history, so you can peek back and get consistent snapshots within the last hour, 6 hours, 24 hours, etc). TTL seems perfect for this, but not if it'll chomp the current version of cells too! :)

> When I was talking to someone the other day about the current TTL policy, he was like "WTF, who would want that, it eats your data?"

I don't think anyone is well served by that kind of shallow analysis.

The TTL feature was introduced for the convenience of having the system automatically garbage collect transient data. If you set a TTL on a column family, you are telling the system that the data shall expire after that interval elapses, that the data is only useful for the configured time period. If the data should not actually be considered transient, then configuring a TTL is the wrong thing to do -- at least currently.

> "TTL except for most recent"

HBASE-4071 is a useful and good idea.

Best regards,- AndyProblems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)>________________________________>From: Ian Varley <[EMAIL PROTECTED]>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>>Sent: Saturday, August 13, 2011 8:24 PM>Subject: Re: TTL for cell values>>So, what you're saying is:>>http://lmgtfy.com/?q=hbase+ttl+remove+all+versions+except+most+recent>>:)>>I like the idea of making this pluggable (via the coprocessor framework, or otherwise). But I also think this is a fundamental enough policy option that making it hard-coded might be a good idea. When I was talking to someone the other day about the current TTL policy, he was like, "WTF, who would want that, it eats your data?". There's no such thing as a "keep 0 versions" option, and thus no way to accidentally lose your most current data using that approach. But with the TTL version there is, which is (IMO) counter-intuitive for those coming from an RDBMS background.>>Commented thusly in the JIRA. :)>>Ian>>On Aug 13, 2011, at 8:00 PM, lars hofhansl wrote:>>Hey Ian, (how are things :)>>I just stumbled across https://issues.apache.org/jira/browse/HBASE-4071.>>-- Lars>>>________________________________>From: Ian Varley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>Sent: Saturday, August 13, 2011 6:51 PM>Subject: TTL for cell values>>Hi all,>>Quick clarification on TTL for cells. The concept makes sense (instead of "keep 3 versions" you say "keep versions more recent than time T"). But, if there's only 1 value in the cell, and that value is older than the TTL, will it also be deleted?>>If so, has there ever been discussion of a "TTL except for most recent" option? (i.e. you want the current version to be permanently persistent, but also want some time-based range of version history, so you can peek back and get consistent snapshots within the last hour, 6 hours, 24 hours, etc). TTL seems perfect for this, but not if it'll chomp the current version of cells too! :)>>Thanks!>Ian>>>>

> "I don't think anyone is well served by that kind of shallow analysis."You're right, Andy; sorry if it came off sounding flip. My point was simply that the idea of a persistent data store with a configuration setting that makes the most current version of your data disappear without an explicit delete is very counter-intuitve for traditional database folks like me. Durability is the first, most inviolate rule, and this setting subverts it in a way that is (at least for me) not obvious at first, and differs drastically from the max versions setting. Maybe my confusion was due to the fact that I was looking for specific behavior (HBASE-4071, essentially). I totally see your point, though; putting it the way I did makes for a rather alarming pull quote. :(

I'm not at all suggesting we should alter the existing behavior (as if that were even possible at this point); this is a useful setting for data that's basically just a cache. But this is an area where the road from RDBMS to HBase might be a little bumpy for folks, and adding a new option would also have the advantage of making it even more clear what TTL is for.

>> When I was talking to someone the other day about the current TTL policy, he was like "WTF, who would want that, it eats your data?"> > I don't think anyone is well served by that kind of shallow analysis. > > The TTL feature was introduced for the convenience of having the system automatically garbage collect transient data. If you set a TTL on a column family, you are telling the system that the data shall expire after that interval elapses, that the data is only useful for the configured time period. If the data should not actually be considered transient, then configuring a TTL is the wrong thing to do -- at least currently.> >> "TTL except for most recent"> > HBASE-4071 is a useful and good idea.> > Best regards,> > > - Andy> > > Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)> > >> ________________________________>> From: Ian Varley <[EMAIL PROTECTED]>>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>>> Sent: Saturday, August 13, 2011 8:24 PM>> Subject: Re: TTL for cell values>> >> So, what you're saying is:>> >> http://lmgtfy.com/?q=hbase+ttl+remove+all+versions+except+most+recent>> >> :)>> >> I like the idea of making this pluggable (via the coprocessor framework, or otherwise). But I also think this is a fundamental enough policy option that making it hard-coded might be a good idea. When I was talking to someone the other day about the current TTL policy, he was like, "WTF, who would want that, it eats your data?". There's no such thing as a "keep 0 versions" option, and thus no way to accidentally lose your most current data using that approach. But with the TTL version there is, which is (IMO) counter-intuitive for those coming from an RDBMS background.>> >> Commented thusly in the JIRA. :)>> >> Ian>> >> On Aug 13, 2011, at 8:00 PM, lars hofhansl wrote:>> >> Hey Ian, (how are things :)>> >> I just stumbled across https://issues.apache.org/jira/browse/HBASE-4071.>> >> -- Lars>> >> >> ________________________________>> From: Ian Varley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>> Sent: Saturday, August 13, 2011 6:51 PM>> Subject: TTL for cell values>> >> Hi all,>> >> Quick clarification on TTL for cells. The concept makes sense (instead of "keep 3 versions" you say "keep versions more recent than time T"). But, if there's only 1 value in the cell, and that value is older than the TTL, will it also be deleted?>> >> If so, has there ever been discussion of a "TTL except for most recent" option? (i.e. you want the current version to be permanently persistent, but also want some time-based range of version history, so you can peek back and get consistent snapshots within the last hour, 6 hours, 24 hours, etc). TTL seems perfect for this, but not if it'll chomp the current version of cells too! :)

On Sunday, August 14, 2011, Ian Varley <[EMAIL PROTECTED]> wrote:>> "I don't think anyone is well served by that kind of shallow analysis.">>> You're right, Andy; sorry if it came off sounding flip. My point wassimply that the idea of a persistent data store with a configuration settingthat makes the most current version of your data disappear without anexplicit delete is very counter-intuitve for traditional database folks likeme. Durability is the first, most inviolate rule, and this setting subvertsit in a way that is (at least for me) not obvious at first, and differsdrastically from the max versions setting. Maybe my confusion was due to thefact that I was looking for specific behavior (HBASE-4071, essentially). Itotally see your point, though; putting it the way I did makes for a ratheralarming pull quote. :(>> I'm not at all suggesting we should alter the existing behavior (as ifthat were even possible at this point); this is a useful setting for datathat's basically just a cache. But this is an area where the road from RDBMSto HBase might be a little bumpy for folks, and adding a new option wouldalso have the advantage of making it even more clear what TTL is for.>> Ian>>> On Aug 13, 2011, at 11:28 PM, "Andrew Purtell" <[EMAIL PROTECTED]>wrote:>>>> When I was talking to someone the other day about the current TTLpolicy, he was like "WTF, who would want that, it eats your data?">>>> I don't think anyone is well served by that kind of shallow analysis.>>>> The TTL feature was introduced for the convenience of having the systemautomatically garbage collect transient data. If you set a TTL on a columnfamily, you are telling the system that the data shall expire after thatinterval elapses, that the data is only useful for the configured timeperiod. If the data should not actually be considered transient, thenconfiguring a TTL is the wrong thing to do -- at least currently.>>>>> "TTL except for most recent">>>> HBASE-4071 is a useful and good idea.>>>> Best regards,>>>>>> - Andy>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein(via Tom White)>>>>>>> ________________________________>>> From: Ian Varley <[EMAIL PROTECTED]>>>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>>>> Sent: Saturday, August 13, 2011 8:24 PM>>> Subject: Re: TTL for cell values>>>>>> So, what you're saying is:>>>>>> http://lmgtfy.com/?q=hbase+ttl+remove+all+versions+except+most+recent>>>>>> :)>>>>>> I like the idea of making this pluggable (via the coprocessor framework,or otherwise). But I also think this is a fundamental enough policy optionthat making it hard-coded might be a good idea. When I was talking tosomeone the other day about the current TTL policy, he was like, "WTF, whowould want that, it eats your data?". There's no such thing as a "keep 0versions" option, and thus no way to accidentally lose your most currentdata using that approach. But with the TTL version there is, which is (IMO)counter-intuitive for those coming from an RDBMS background.>>>>>> Commented thusly in the JIRA. :)>>>>>> Ian>>>>>> On Aug 13, 2011, at 8:00 PM, lars hofhansl wrote:>>>>>> Hey Ian, (how are things :)>>>>>> I just stumbled across https://issues.apache.org/jira/browse/HBASE-4071.>>>>>> -- Lars>>>>>>>>> ________________________________>>> From: Ian Varley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>> To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>>>> Sent: Saturday, August 13, 2011 6:51 PM>>> Subject: TTL for cell values>>>>>> Hi all,>>>>>> Quick clarification on TTL for cells. The concept makes sense (insteadof "keep 3 versions" you say "keep versions more recent than time T"). But,if there's only 1 value in the cell, and that value is older than the TTL,will it also be deleted?>>>>>> If so, has there ever been discussion of a "TTL except for most recent"option? (i.e. you want the current version to be permanently persistent, butalso want some time-based range of version history, so you can peek back andget consistent snapshots within the last hour, 6 hours, 24 hours, etc). TTLseems perfect for this, but not if it'll chomp the current version of cellstoo! :)

I am slightly confused now. Time to live is used in networking , after nhops drop this packet. Also used I'm memcache , expire this data n secondsafter insert.

I do not know of any specific ttl features in rdbms so I do not understandwhy someone would expect ttl to he permanently durable.

"I am slightly confused now. Time to live is used in networking , after nhops drop this packet. Also used I'm memcache , expire this data n secondsafter insert. I do not know of any specific ttl features in rdbms so I do not understandwhy someone would expect ttl to he permanently durable."

Edward, my mistake was originally assuming that the TTL applied only to *old* versions of a cell (I.e. not the most recent one). It was a misunderstanding on my part, based on the fact that there *are* no TTL features in an RDBMS (you only get rid of the current value by issuing an explicit delete).

HBase is light years ahead of an RDBMS in the way it explicitly handles the time dimension; I just wasn't expecting this facet of that behavior. It's part of the mindset shift you have to go through coming from a database world to a NoSQL world; you can treat parts of your store as a transient cache if you want to (which is useful in all kinds of situations). Just needed to expand my brain to consider that ... :)

> It's part of the mindset shift you have to go through coming from a database world to a NoSQL world

This is useful. If you have more insights like this Ian and care to share them, I think we would be really interested to hear them.

Best regards, - AndyProblems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)>________________________________>From: Ian Varley <[EMAIL PROTECTED]>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>>Sent: Sunday, August 14, 2011 8:36 AM>Subject: Re: TTL for cell values>>"I am slightly confused now. Time to live is used in networking , after n>hops drop this packet. Also used I'm memcache , expire this data n seconds>after insert. I do not know of any specific ttl features in rdbms so I do not understand>why someone would expect ttl to he permanently durable.">>Edward, my mistake was originally assuming that the TTL applied only to *old* versions of a cell (I.e. not the most recent one). It was a misunderstanding on my part, based on the fact that there *are* no TTL features in an RDBMS (you only get rid of the current value by issuing an explicit delete).>>HBase is light years ahead of an RDBMS in the way it explicitly handles the time dimension; I just wasn't expecting this facet of that behavior. It's part of the mindset shift you have to go through coming from a database world to a NoSQL world; you can treat parts of your store as a transient cache if you want to (which is useful in all kinds of situations). Just needed to expand my brain to consider that ... :)>>Ian>>>On Aug 14, 2011, at 7:30 AM, "Edward Capriolo" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:>>I am slightly confused now. Time to live is used in networking , after n>hops drop this packet. Also used I'm memcache , expire this data n seconds>after insert.>>I do not know of any specific ttl features in rdbms so I do not understand>why someone would expect ttl to he permanently durable.>>>

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext