As we're starting to consider 1.5.2 and 1.6.1 coming out in the near future, I want to revisit a discussion[1] I started at the end of April regarding the "testing burden" that is currently set forth in our release document[2].

What I'm proposing is to modify the language of the release document to be explicit about the amount of testing needed. For bug-fix, "minor" releases (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and randomwalk (with and without agitation) will be clearly defined as "may" instead of "should" or "must" language. If the resources are available, it is recommended that some longer, multi-process/node test is run against the release candidate; however, it is not required and should not prevent us from making the minor release.

I will also include language that strongly recommends that the changes included in the "minor" release be vetted/reviewed as a way to mitigate the risk of shipping new regressions.

I am not recommending that the language be changed for "major" releases (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new features or internal changes.

Unless someone informs me otherwise, I will treat this as a normal lazy-consensus approval. Assuming we move closer to "proper" semantic versioning for 2.0.0, I believe these updated guidelines will change again. I do however think there is merit in making this change now so that we can get the good bugs that we've fixed out to our users.

Let me know what you think. I will wait, at least, the prescribed three days before changing any thing.

-1 I hesitate to step into this discussion because I can't also stepup and do the long-term testing even as I recommend that it must bedone. There are at least four companies supporting Accumulo andcontributing back to the project. Surely one of those companies cansupply the resources to continue the existing test regimen? Is theresome concern that those resources won't be available for the nextrelease cycle?

I don't know Josh's concerns, but the concern for me is both resources andtime. No matter how much resources we have, it is still not infinite, andI'd rather we focus our testing efforts on the changeset between theprevious release and the minor/bugfix release, rather than spend resourcesand time on all the exhaustive general testing, which mostly exercises codethat has not changed.

For instance, do we really need 72-hours of continuous ingest on a largecluster to release a bugfix which affects the shell?If the long running tests are what is necessary to exercise the changeset,that makes sense, but otherwise, no.

I realize we're past window here, but a few things:1) I'm concerned about changing our testing standards prior to getting somepractice in on holding to tighter bounds on what can go into a particularkind of fix number. However, I generally like the idea of just relying onpeople voting -1 for insufficient testing.2) We really could have used a [VOTE] on changing 1.x versions to usebugfix for the last number. It would be an opportunity to formally adoptour release governance docs into the PMC bylaws by reference.3) As a point of principle, David's veto was completely valid as it stood.While we are individuals it's perfectly reasonable for the PMC to setrequirements on what goes into a release we sign off on, even if the PMCmembers may not always have ready access to the resources needed to meetthat standard.

There's no reason David should have to volunteer resources to back up hisveto, especially when it was merely calling for a continuation of thestandard we already had set.

4) While I'm not in a position to obligate Cloudera, the team I'm oncurrently has been allocated sufficient resources to meet the currenttesting standards for a release and I have no reason to believe we won'thave said resources when future release windows happen.

It's ok. I only remembered about this early today.I don't see this as a huge issue since everyone seemed to be in agreement on it. Writing it down for reference would be good for future discussions on the subject.That's why I said "Unless you are willing to supply the funds to pay to use some resources, I don't feel like this is a valid -1." If he, or anyone, is willing provide general resources for testing, that's a different story. Given his response, I assume that is not the case.This would require time from you, Mike or Bill H, yes? While having some dedicated resources is nice, I'm worried about relying on other people (who have their own obligations) to fulfill the release manager's obligations.

I think that gets into the details which the release manager can coordinate and other devs can express their concerns via the normal release voting process.

But that's the opposite of what I just said. The opposition and the abilityto find funds are not strongly coupled.

The lack of funds would hopefully be a convincing argument to try to swaysomeone that we should lower the testing barrier, but they aren't alegitimate excuse for invalidating their vote. What if Hypothetical Davidwanted to do fund raising to get resources together? Would you decide on adeadline that would allow his vote to be legitimate or not?

The basis for a veto need merely be technical. "We've done this level oftesting before and it protects our users. We should continue doing it." isa perfectly reasonable justification. (I admit I am taking some liberty inmaking parts of David's previous concern more explicit)

BTW, situations like this are a part of why Majority Vote for governancedecisions are my preference. While I fully agree with David's ability toveto I also agree that the community should be able to override him if nofunding source could be found.

The release manager's only responsibility is to ensure the testing getsdone, not to do the testing themselves. That's why we're a community: so wecan rely on each other.

If a particular release manager needs help making sure the coveragehappens, they should just ask for that help when making the release plan.Mike, Bill, and I are all capable of getting approval to block out time insupport of things that are good for Apache Accumulo.

(I am not actually sure it would require time from Mike, Bill, or me;circumstances happen and "it depends." It's certainly easier for Mike,Bill, or me to do it.)

Oh it is. I apologize, I misread your initial statement.I wouldn't have a problem with anyone trying to raise funds for test resources, within reason. But, unless someone is actually planning to do this, it probably isn't much more than a hypothetical argument that we need to spend time discussing.

I've update the language on the release guidelines on the staged site[1] for what was discussed here.

Please let me know if there are any problems with it, otherwise I'll push the changes to production later today. If you miss this message before I push to production, please feel free to continue the discussion and we can revise the language and make sure we're all happy.

as a follow up, can we agree on a label or add a flag to jiras that can beused to signal when we think a particular issue introduces changes thatwarrant the higher level of testing scrutiny?On Fri, Jun 27, 2014 at 12:56 PM, Josh Elser <[EMAIL PROTECTED]> wrote:

There's a label "needs-test" that we don't already use for anything. Wecould use this to mean "needs more than bugfix level testing."

I started looking at what it takes to add a field, but it's not obvious atfirst glance. If we figure it out, we could add a "test level" field thatsays "bugfix", "minor", "major" to indicate how much testing we need. Evenif we currently don't define all three levels, where ever we explain theuse of the field we could conflate as we desire.

On Fri, Jun 27, 2014 at 4:34 PM, Christopher <[EMAIL PROTECTED]> wrote:oh, yeeeeeessss. that would be nice. that would give us a better idea oftesting coverage without requiring explicit coordination pre-RC.

Sean

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext