On 10 February 2014 09:24, Stack <[EMAIL PROTECTED]> wrote:love to see how that goes on -it looks good but I've not seen it inproduction yet. I do note it has a NoSQL outputter, so there are goodopportunities for recursion

+ maybe make the hconnection output a bit less verbose when ZK isn't liveor HBase still coming up; it does generate a fair amount of log data+ light pastel purple colour scheme.

CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

+ a clearly defined compatibility strategy between patch, minor, and major versions+ known coprocessor interfaces. For example right now a coprocessor can use everything on HRegion. We need to distill what's useful into an interface.+ (wishes for ponies and unicorns, probably not going to happen) better ops supports, such as sample puppet/chef/etc scripts

We started this with Environment and the HTable wrapper as an example ofhow to wrap an interface for CPs. At the end of the day we can't stop acoprocessor from accessing internal objects and calling their methodsdirectly until HBASE-4047. (Maybe that makes the 1.0 list?)

Related, paring down the coprocessor interfaces to only intercept RPC basedactions and move everything else to plugins.

True. But we can clearly annotate what is public and what is not.In HRegion we'd have to do it per method. It's probably easier to extract the public interface out into a Interface. (similar to Store and HStore)HBASE-4207 solves a slightly different problem. Here I want to have access to some HBase internals for performance, but I want to know what is public and stable and what I should not touch (thinking about things like Phoenix).

On Mon, Feb 10, 2014 at 4:42 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:We started this with Environment and the HTable wrapper as an example ofhow to wrap an interface for CPs. At the end of the day we can't stop acoprocessor from accessing internal objects and calling their methodsdirectly until HBASE-4047. (Maybe that makes the 1.0 list?)

Related, paring down the coprocessor interfaces to only intercept RPC basedactions and move everything else to plugins.

We need to revisit which interfaces should be marked as real "private"because coprocessors allow custom applications to accesses internalstates. Current private interface such as KeyValue, WALEdit, Hregion,HLogKey, Store etc is may used by custom coprocessor applications so anypublic method signature change in those "private" interfaces possibliybreaks custom coprocessor implementations.

-Jeffrey

On 2/10/14 4:59 PM, "lars hofhansl" <[EMAIL PROTECTED]> wrote:CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

I mentioned this in phoenix-dev that we need to repurpose coprocessors, andjust give them put / get kind of injection points, but not allowco-processors to tap into log / compaction, etc. Those will be betterserved by explicitly defined plugins (like the recent StorageEngine, HLog,etc) that we explicitly allow injecting code with some API stability(though probably still evolving at a fast pace).

It would be good to have an idea of what classes / methods, phoenix andsimilar depend on HBase right now.

Lily's indexer uses hbase private apis in the replication portion of code. we may want to expose and make promises on the replication api/wire tomake their lives easier and to enable cross version replication for futurehbases.

Thanks Stack for bringing this up. I actually want to volunteer as an RMfor 1.0 release. We can do a 0.99 release first as a dev release, then turn0.99.x into 1.0 as we did in 0.95 -> 0.96 releases.

As for timing, we can start by filing some jiras for the discussion itemsand setting the target versions for those to see where we are at. 0.98 willbe out soon, and we want to make 1.0 sooner rather than later. Also if wecan nail the release before HBaseCon that would be good with users.

Coprocessors were and are a means for internal server extension by mixin. The original problem we solved was needing to subclass HRegionServer and other classes to extend core HBase functions, but having more than one otherwise orthogonal extension that users want to use. Now we can mix in multiple extensions with a framework that has some simple rules for cooperation between the extensions.

We return to the earlier state of affairs with modules. Sure, we can plug in an alternate behavior with a module that subclasses and extends the default, say flush strategy, but we can't then instantiate multiple modules into the same slot, both subclassing the same base but doing different things.

I agree the ability to compose coprocessors in order to extend behavior isa key capability that we should not throw out.

I think the current Observer APIs could probably do with a bit ofreorganization to make them a little more accessible and comprehensible. Ithink there is also an emerging need to see if we can define some subset ofthese APIs that we can stabilize for easier public consumption, whilekeeping the rest of the APIs free to evolve as needed as HBase internalschange (since these are an extension mechanism for internal behaviors). I'm not sure we've really seen enough commonality emerge yet to say whatthose APIs are though. We could try to define the public subset as thoseinvolved in client requests, but flush and compaction, for example, canalso be triggered by client requests. And my own use of coprocessor APIslately has been focused on overriding the flush and compaction behaviors,not on client requests.

I think the best place to start is by breaking up some of the current APIs,grouping them around behaviors or areas of functionality. Whether we callsome of these "coprocessors" and others "plugins" is a question ofbranding. I do think it's important to figure out which we can stabilizeand offer longer term contracts for. But whatever we call them, I stronglyagree that we should maintain the "mixin" / composition approach and notreturn to a simple fixed inheritance scheme.

On Wed, Feb 12, 2014 at 12:46 PM, Gary Helmling <[EMAIL PROTECTED]> wrote:I've always considered coprocessors to be the "kernel modules" of the HBaseworld. They give you way more power than user-space programming, but comewith the cautions that if you make a mistake, you'll crash your wholesystem or trigger unexpected behavior.

Given that, I don't think we should really be spending too much effort oncoprocessor API stability. If we make this a requirement, it can hamper theability of the HBase core developers to make good changes which reallyimprove the system. I don't think we're at the level of maturity as aproject where this is the right tradeoff, as of yet.

I do think we should seek to keep the interfaces stable through *patch*level releases -- a bug fix shouldn't break a coprocessor API. But betweenminor releases that add new features, it seems like an unnecessaryrestriction.

This is no coincidence, Linux kernel modules (specifically the securityones, and what enables them) were some direct inspiration. :-)

I may have said this before but I think stable interfaces for coprocessorscomes into play only once we are doing HBASE-4047, and even then I wouldexpect an eventing model where the vocabulary evolves from release torelease.On Wed, Feb 12, 2014 at 1:22 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:Best regards,

+ HBASE-8763 Combine MVCC and SeqIdCurrently replication is broken on same version updates in the followingscenarios: when a region move or RS get restarted in the source cluster,replicated changes may come out of order to the receiving RS. There areother cases we need to keep the ordering of puts. This JIRA can be used tofix those above issues.

+ Replicated Meta Regions for Distributed Region Location Lookup(I haven'tcut a JIRA for it but I've been thinking this for a while)On 2/10/14 4:42 PM, "lars hofhansl" <[EMAIL PROTECTED]> wrote:CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.