This is the third candidate for release of Apache Kafka 0.8.0. Thisrelease candidate artifacts changed in regards tohttps://issues.apache.org/jira/browse/KAFKA-1126 along with updating therelease notes to only have resolved - fixed issues and the KEY file toverify signature again for this vote to be the latest link.

The following are very minor issues that I think don't matter at all,but we can keep in mind for future releases:- won't fix issues still show up in the release notes (the go client jiras for e.g.,)- Inhttp://people.apache.org/~joestein/kafka-0.8.0-candidate3/kafka_2.8.0-0.8.0.tar.gz we package the perf scripts (for producer/consumer) but don't include the perf jar. So maybe those scripts should be removed (and same for run-rat.sh). Can you file a jira for this?

Thanks David, I am going to have to throw down a -1 on the binaryartifacts, your right we need LICENSE and NOTICE in the binary distro alsothe JARs we are putting up into Maven (they are supposed to be in theMANIFEST and their not). I created a ticket for the former but haven't yetfor the latter ... haven't figured out how to-do that with SBT yet.

So, before I propose a new VOTE I think fixing the LICENSE and NOTICE forthe binary distro is going to take 13 seconds to fix. Getting the LICENSEand NOTICE into the jars that are published is going to take a littlelonger. I will start a discussion on the mailing list.

> +0 overall. The quality of the build seems good, but I have a few minor> nits/questions.>> 1) java.lang.NoClassDefFoundError in ./bin/kafka-simple-consumer-> perf-test.sh>> $ ./bin/kafka-simple-consumer-perf-test.sh> Exception in thread "main" java.lang.NoClassDefFoundError: kafka/perf/> SimpleConsumerPerformance> Caused by: java.lang.ClassNotFoundException: kafka.perf.> SimpleConsumerPerformance> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)> at java.security.AccessController.doPrivileged(Native Method)> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)>> looks like the perf classes are not included. Maybe this script should be> removed? Or we should include the perf sub-project.>> 2) No README or INSTALL in the binary dist>> 3) Missing LICENSE and NOTICE at the top level, aren't these required for> an Apache release?>> 4) Artifact name - not sure on Apache's policy, but should it be> "apache-kafka"?>> Regarding the actual build, I started up with the bin script, produced a> few messages, consumed a few, and then ran the integration tests from> kafka-python against it. Everything seems good here.>> -David>>> On 11/8/13 3:56 PM, Joe Stein wrote:>>> This is the third candidate for release of Apache Kafka 0.8.0. This>> release candidate artifacts changed in regards to>> https://issues.apache.org/jira/browse/KAFKA-1126 along with updating the>> release notes to only have resolved - fixed issues and the KEY file to>> verify signature again for this vote to be the latest link.>>>> Release Notes for the 0.8.0 release>> http://people.apache.org/~joestein/kafka-0.8.0->> candidate3/RELEASE_NOTES.html>>>> *** Please download, test and vote by Wednesday November, 13th, 12pm PDT>>>> Kafka's KEYS file containing PGP keys we use to sign the release:>> http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5 and>> sha1>> checksum>>>> * Release artifacts to be voted upon (source and binary):>> http://people.apache.org/~joestein/kafka-0.8.0-candidate3/>>>> * Maven artifacts to be voted upon prior to release:>> https://repository.apache.org/content/groups/staging/>>>> (i.e. in sbt land this can be added to the build.sbt to use Kafka>> resolvers += "Apache Staging" at ">> https://repository.apache.org/content/groups/staging/">> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0">> )>>>> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=>> bc24fb7c0f48c36bf9073d384eb87213644648d4>>>> /*******************************************>> Joe Stein>> Founder, Principal Consultant>> Big Data Open Source Security LLC

I'm getting this with console consumer in the binary package whichseems to be because it was built under Java 7. Can we stick to 6?

Exception in thread "main" java.lang.UnsupportedClassVersionError: kafka/javaapi/consumer/ConsumerConnector : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178) at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)

Other minor comments:- I don't think the SBT license needs to be included in the binary package (at the end of LICENSE).- Would prefer that the release notes remove jiras that were resolved as won't fix - e.g., the go client jiras.

Seems like most people are verifying the src, so I'll pick on the binaries and Maven stuff ;)

A few problems I see:

There are some vestigial Git files in the src download: an empty .git and .gitignore

In the source download, I see the SBT license in LICENSE which seems correct (since we distribute an SBT binary), but in the binary download I see the same license. Don't we need the Scala license (http://www.scala-lang.org/license.html) in the binary distribution?

I create a simple Ant+Ivy project to test resolving the artifacts published to Apache staging repo: https://github.com/mumrah/kafka-ivy. This will fetch Kafka libs from the Apache staging area and other things from Maven Central. It will fetch the jars into lib/ivy/{conf} and generate a report of the dependencies, conflicts, and licenses into ivy-report. Notice I had to add three exclusions to get things working. Maybe we should add these to our pom?

I think I'll have to -1 the release due to the missing Scala license in the binary dist. We should check the other licenses as well (see ivy-report from my little Ant project).

Talking about binary release, do we really need to include bin/run-rat.sh?As far as I understand it is only used to bring licenses to the scene andquite redundant for already baked release.

Next, I not quite sure, but probably it makes sense to droplibs/scala-compiler.jar -- kafka do not perform compilations during runtimeand this step will trim some fat from the resulting release (from 17 mbdown to 9.5 mb*).

I managed to satisfy maven with only two exclusions, but yes, it would begood to see them in original pom.

* by the way using the best possible compression method (-9 instead ofdefault -6) + drop of compiler lib gave me the very same result -- 9.5 Mb2013/12/2 David Arthur <[EMAIL PROTECTED]>

> Seems like most people are verifying the src, so I'll pick on the binaries> and Maven stuff ;)>> A few problems I see:>> There are some vestigial Git files in the src download: an empty .git and> .gitignore>> In the source download, I see the SBT license in LICENSE which seems> correct (since we distribute an SBT binary), but in the binary download I> see the same license. Don't we need the Scala license (> http://www.scala-lang.org/license.html) in the binary distribution?>> I create a simple Ant+Ivy project to test resolving the artifacts> published to Apache staging repo: https://github.com/mumrah/kafka-ivy.> This will fetch Kafka libs from the Apache staging area and other things> from Maven Central. It will fetch the jars into lib/ivy/{conf} and generate> a report of the dependencies, conflicts, and licenses into ivy-report.> Notice I had to add three exclusions to get things working. Maybe we should> add these to our pom?>> I think I'll have to -1 the release due to the missing Scala license in> the binary dist. We should check the other licenses as well (see ivy-report> from my little Ant project).>> -David>> On 11/26/13 5:34 PM, Joe Stein wrote:>>> This is the fifth candidate for release of Apache Kafka 0.8.0. This>> release candidate is now built from JDK 6 as RC4 was built with JDK 7.>>>> Release Notes for the 0.8.0 release>> http://people.apache.org/~joestein/kafka-0.8.0->> candidate5/RELEASE_NOTES.html>>>> *** Please download, test and vote by Monday December, 2nd, 12pm PDT>>>> Kafka's KEYS file containing PGP keys we use to sign the release:>> http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5 and>> sha1>> checksum>>>> * Release artifacts to be voted upon (source and binary):>> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/>>>> * Maven artifacts to be voted upon prior to release:>> https://repository.apache.org/content/groups/staging/>>>> (i.e. in sbt land this can be added to the build.sbt to use Kafka>> resolvers += "Apache Staging" at ">> https://repository.apache.org/content/groups/staging/">> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0">> )>>>> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=>> 2c20a71a010659e25af075a024cbd692c87d4c89>>>> /*******************************************>> Joe Stein>> Founder, Principal Consultant>> Big Data Open Source Security LLC>> http://www.stealth.ly>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>>> ********************************************/>>>>>

General future thought comment first: lets be careful please to raisingissues as show stoppers that have been there previously (especially ifgreater than one version previous release back also has the problem) andcan get fixed in a subsequent release and is only now more pressing becausewe know about them... seeing something should not necessarily always createpriority (sometimes sure, of course but not always that is not the best wayto manage changes). The VOTE thread should be to artifacts and what we arereleasing as proper and correct per Apache guidelines... and to make surethat the person doing the release doesn't do something incorrect ... likeusing the wrong version of JDK to build =8^/. If we are not happy withrelease as ready to ship then lets not call a VOTE and save the prolongedweeks that drag out with so many release candidates. The community suffersfrom this.

ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...hopefully a few more hours for other folks to comment and discuss theissues you raised with my $0.02852425 included below and follow-ups as theybecome necessary... I am also out of pocket in a few hours until tomorrowmorning so if it passed I would not be able to publish and announce or iffailed look towards RC6 anyways =8^)

> Seems like most people are verifying the src, so I'll pick on the binaries> and Maven stuff ;)>> A few problems I see:>> There are some vestigial Git files in the src download: an empty .git and> .gitignore>

Ok, I can do a better job with 0.8.1 but I am not sure this is verydifferent than beta1 and not necessarily a show stopper for 0.8.0 requiringanother release candidate, is it? I think updating the release docs andrmdir .git after the rm -fr and rm .gitignore moving forward makes sense.>> In the source download, I see the SBT license in LICENSE which seems> correct (since we distribute an SBT binary), but in the binary download I> see the same license. Don't we need the Scala license (> http://www.scala-lang.org/license.html) in the binary distribution?>

I fixed this already not only in the binary releasehttps://issues.apache.org/jira/browse/KAFKA-1131 but also in the JAR filesthat are published to Mavenhttps://issues.apache.org/jira/browse/KAFKA-1133are you checking fromhttp://people.apache.org/~joestein/kafka-0.8.0-candidate5/ because I justdownloaded again and it looks alright to me. If not then definitely thisRC should be shot down because it does not do what we are saying it isdoing.. but if it is wrong can you be more specific and create a JIRA withthe fix because I thought I got it right already... but if not then letsget it right because that is why we pulled the release in RC3>> I create a simple Ant+Ivy project to test resolving the artifacts> published to Apache staging repo: https://github.com/mumrah/kafka-ivy.> This will fetch Kafka libs from the Apache staging area and other things> from Maven Central. It will fetch the jars into lib/ivy/{conf} and generate> a report of the dependencies, conflicts, and licenses into ivy-report.> Notice I had to add three exclusions to get things working. Maybe we should> add these to our pom?>

I don't think this is a showstopper is it? can't this wait for 0.8.1 andnot hold up the 0.8.0 release?

I didn't have this issue with java maven pom or scala sbt so maybesomething more ivy ant specific causing this? folks use gradle too so Iexpect some feedback at some point to that working or not perhaps in 0.8.1or even 0.9 we can try to cover every way everyone uses and make sure theyare all good to go moving forward... perhaps even some vagrant, docker,puppet and chef love too (which I can contribute if folks are interested)=8^)

In any case can you create a JIRA and throw a patch up on it please,thanks! IMHO this is for 0.8.1 though ... what are thoughts here...

it would break my heart to have lots of binding +1 votes and 2 non-bindingvotes one +1 and one -1, I still haven't cast my vote yet was hopingeveryone would get their voices and everything in before calling the VOTEclosed or canceled. I really don't mind preparing a release candidate 6that is not the issue at all but I think we need to be thoughtful aboutusing the release candidates to fixe things that should be fixed and partof the releases themselves where the release candidates are to make surethat the preparation of the build is not wrong (like it was in RC4 where Iused JDK 7 by accident).

I think we should maintain a wiki describing the release process in detail,so we save the turnaround time on a release. We can have a VOTE thread toagree on the release guidelines and follow it. Having said that, it isworth having the correct .pom file at the very least, since the release isnot very useful if people cannot consume it without pain.

> General future thought comment first: lets be careful please to raising> issues as show stoppers that have been there previously (especially if> greater than one version previous release back also has the problem) and> can get fixed in a subsequent release and is only now more pressing because> we know about them... seeing something should not necessarily always create> priority (sometimes sure, of course but not always that is not the best way> to manage changes). The VOTE thread should be to artifacts and what we are> releasing as proper and correct per Apache guidelines... and to make sure> that the person doing the release doesn't do something incorrect ... like> using the wrong version of JDK to build =8^/. If we are not happy with> release as ready to ship then lets not call a VOTE and save the prolonged> weeks that drag out with so many release candidates. The community suffers> from this.>> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...> hopefully a few more hours for other folks to comment and discuss the> issues you raised with my $0.02852425 included below and follow-ups as they> become necessary... I am also out of pocket in a few hours until tomorrow> morning so if it passed I would not be able to publish and announce or if> failed look towards RC6 anyways =8^)>> /*******************************************> Joe Stein> Founder, Principal Consultant> Big Data Open Source Security LLC> http://www.stealth.ly> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>> ********************************************/>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]> wrote:>> > Seems like most people are verifying the src, so I'll pick on the> binaries> > and Maven stuff ;)> >> > A few problems I see:> >> > There are some vestigial Git files in the src download: an empty .git and> > .gitignore> >>> Ok, I can do a better job with 0.8.1 but I am not sure this is very> different than beta1 and not necessarily a show stopper for 0.8.0 requiring> another release candidate, is it? I think updating the release docs and> rmdir .git after the rm -fr and rm .gitignore moving forward makes sense.>>> >> > In the source download, I see the SBT license in LICENSE which seems> > correct (since we distribute an SBT binary), but in the binary download I> > see the same license. Don't we need the Scala license (> > http://www.scala-lang.org/license.html) in the binary distribution?> >>> I fixed this already not only in the binary release> https://issues.apache.org/jira/browse/KAFKA-1131 but also in the JAR files> that are published to Maven> https://issues.apache.org/jira/browse/KAFKA-1133are you checking from> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/ because I just> downloaded again and it looks alright to me. If not then definitely this> RC should be shot down because it does not do what we are saying it is> doing.. but if it is wrong can you be more specific and create a JIRA with> the fix because I thought I got it right already... but if not then lets> get it right because that is why we pulled the release in RC3>>> >> > I create a simple Ant+Ivy project to test resolving the artifacts> > published to Apache staging repo: https://github.com/mumrah/kafka-ivy.> > This will fetch Kafka libs from the Apache staging area and other things> > from Maven Central. It will fetch the jars into lib/ivy/{conf} and> generate> > a report of the dependencies, conflicts, and licenses into ivy-report.

which very closely mirrors what David was talking about with ivy as well...I didn't really think much of it just a matter of XML we can document(there is actually no using maven documentation on the site at all weshould correct that in any case TBD post release) but if folks find it tobe a pain then we should definitely fix it for sure. off the top of myhead I don't see how to-do that in the Build.scala but I really don'texpect it to be too difficult to figure out... the question is do we holdit off for 0.8.1 since technically nothing is breaking (like the nullpointer exceptions we had for the bonked pom in beta1 that I shipped tomaven central).

Before canceling the vote can we at least get consensus to what we arecanceling and exactly what fixes should be in RC6 or ... agree to ship RC5and hold whatever is left for 0.8.1

I am totally fine with working on RC6 (actually just cancelled my plans forthe evening because of a whole slew of client work that hit my plate) but Iwant to make sure we have everything covered that everyone that is votingexpects to be in there.

> I think we should maintain a wiki describing the release process in detail,> so we save the turnaround time on a release. We can have a VOTE thread to> agree on the release guidelines and follow it. Having said that, it is> worth having the correct .pom file at the very least, since the release is> not very useful if people cannot consume it without pain.>> Thanks,> Neha>>> On Mon, Dec 2, 2013 at 8:59 AM, Joe Stein <[EMAIL PROTECTED]> wrote:>> > General future thought comment first: lets be careful please to raising> > issues as show stoppers that have been there previously (especially if> > greater than one version previous release back also has the problem) and> > can get fixed in a subsequent release and is only now more pressing> because> > we know about them... seeing something should not necessarily always> create> > priority (sometimes sure, of course but not always that is not the best> way> > to manage changes). The VOTE thread should be to artifacts and what we> are> > releasing as proper and correct per Apache guidelines... and to make sure> > that the person doing the release doesn't do something incorrect ... like> > using the wrong version of JDK to build =8^/. If we are not happy with> > release as ready to ship then lets not call a VOTE and save the prolonged> > weeks that drag out with so many release candidates. The community

On 12/2/13 11:59 AM, Joe Stein wrote:> General future thought comment first: lets be careful please to raising> issues as show stoppers that have been there previously (especially if> greater than one version previous release back also has the problem) and> can get fixed in a subsequent release and is only now more pressing because> we know about them... seeing something should not necessarily always create> priority (sometimes sure, of course but not always that is not the best way> to manage changes). The VOTE thread should be to artifacts and what we are> releasing as proper and correct per Apache guidelines... and to make sure> that the person doing the release doesn't do something incorrect ... like> using the wrong version of JDK to build =8^/. If we are not happy with> release as ready to ship then lets not call a VOTE and save the prolonged> weeks that drag out with so many release candidates. The community suffers> from this.+1 If we can get most of this release preparation stuff automated, then we can iterate on it in a release branch before tagging and voting.> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...> hopefully a few more hours for other folks to comment and discuss the> issues you raised with my $0.02852425 included below and follow-ups as they> become necessary... I am also out of pocket in a few hours until tomorrow> morning so if it passed I would not be able to publish and announce or if> failed look towards RC6 anyways =8^)>> /*******************************************> Joe Stein> Founder, Principal Consultant> Big Data Open Source Security LLC> http://www.stealth.ly> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>> ********************************************/>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>> Seems like most people are verifying the src, so I'll pick on the binaries>> and Maven stuff ;)>>>> A few problems I see:>>>> There are some vestigial Git files in the src download: an empty .git and>> .gitignore>>> Ok, I can do a better job with 0.8.1 but I am not sure this is very> different than beta1 and not necessarily a show stopper for 0.8.0 requiring> another release candidate, is it? I think updating the release docs and> rmdir .git after the rm -fr and rm .gitignore moving forward makes sense.Agreed, not a show stopper.>>>> In the source download, I see the SBT license in LICENSE which seems>> correct (since we distribute an SBT binary), but in the binary download I>> see the same license. Don't we need the Scala license (>> http://www.scala-lang.org/license.html) in the binary distribution?>>> I fixed this already not only in the binary release> https://issues.apache.org/jira/browse/KAFKA-1131 but also in the JAR files> that are published to Maven> https://issues.apache.org/jira/browse/KAFKA-1133are you checking from> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/ because I just> downloaded again and it looks alright to me. If not then definitely this> RC should be shot down because it does not do what we are saying it is> doing.. but if it is wrong can you be more specific and create a JIRA with> the fix because I thought I got it right already... but if not then lets> get it right because that is why we pulled the release in RC3The LICENSE file in both the src and binary downloads includes "SBT LICENSE" at the end. I could be wrong, but I think the src download should include the SBT licnese and the binary download should include the Scala license. Since we have released in the past without proper licensing, it's probably not a huge deal to do it again (but we should fix it).>>> I create a simple Ant+Ivy project to test resolving the artifacts>> published to Apache staging repo: https://github.com/mumrah/kafka-ivy.>> This will fetch Kafka libs from the Apache staging area and other things>> from Maven Central. It will fetch the jars into lib/ivy/{conf} and generateNo I don't think it's a show stopper. But to Neha's point, a painless Maven/Ivy/SBT/Gradle integration is important since this is how most users interface with Kafka. That said, ZooKeeper is what's pulling in these troublesome deps and it doesn't stop people from using ZooKeeper. I can live with this.No clue... maybe? I run into these deps all the time when dealing with ZooKeeper.

Does one -1 kill the vote? I'm mainly trying to be the squeaky wheel here ;) - I'd rather not hold up the release and just fix these issues for 0.8.1.

> Inline:>>> On 12/2/13 11:59 AM, Joe Stein wrote:>>> General future thought comment first: lets be careful please to raising>> issues as show stoppers that have been there previously (especially if>> greater than one version previous release back also has the problem) and>> can get fixed in a subsequent release and is only now more pressing>> because>> we know about them... seeing something should not necessarily always>> create>> priority (sometimes sure, of course but not always that is not the best>> way>> to manage changes). The VOTE thread should be to artifacts and what we>> are>> releasing as proper and correct per Apache guidelines... and to make sure>> that the person doing the release doesn't do something incorrect ... like>> using the wrong version of JDK to build =8^/. If we are not happy with>> release as ready to ship then lets not call a VOTE and save the prolonged>> weeks that drag out with so many release candidates. The community>> suffers>> from this.>>> +1 If we can get most of this release preparation stuff automated, then we> can iterate on it in a release branch before tagging and voting.>> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...>> hopefully a few more hours for other folks to comment and discuss the>> issues you raised with my $0.02852425 included below and follow-ups as>> they>> become necessary... I am also out of pocket in a few hours until tomorrow>> morning so if it passed I would not be able to publish and announce or if>> failed look towards RC6 anyways =8^)>>>> /*******************************************>> Joe Stein>> Founder, Principal Consultant>> Big Data Open Source Security LLC>> http://www.stealth.ly>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>>> ********************************************/>>>>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>>> Seems like most people are verifying the src, so I'll pick on the>>> binaries>>> and Maven stuff ;)>>>>>> A few problems I see:>>>>>> There are some vestigial Git files in the src download: an empty .git and>>> .gitignore>>>>>> Ok, I can do a better job with 0.8.1 but I am not sure this is very>> different than beta1 and not necessarily a show stopper for 0.8.0>> requiring>> another release candidate, is it? I think updating the release docs and>> rmdir .git after the rm -fr and rm .gitignore moving forward makes sense.>>> Agreed, not a show stopper.>>>>>> In the source download, I see the SBT license in LICENSE which seems>>> correct (since we distribute an SBT binary), but in the binary download I>>> see the same license. Don't we need the Scala license (>>> http://www.scala-lang.org/license.html) in the binary distribution?>>>>>> I fixed this already not only in the binary release>> https://issues.apache.org/jira/browse/KAFKA-1131 but also in the JAR>> files>> that are published to Maven>> https://issues.apache.org/jira/browse/KAFKA-1133are you checking from>> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/ because I just>> downloaded again and it looks alright to me. If not then definitely this>> RC should be shot down because it does not do what we are saying it is>> doing.. but if it is wrong can you be more specific and create a JIRA with>> the fix because I thought I got it right already... but if not then lets>> get it right because that is why we pulled the release in RC3>>> The LICENSE file in both the src and binary downloads includes "SBT> LICENSE" at the end. I could be wrong, but I think the src download should> include the SBT licnese and the binary download should include the Scala

I think maybe the 2.8.0 POM is missing some stuff it needs (zkclient, snappy, yammer metrics). And there is a duplicate ZK entry for the POMs >2.8.0

-David

On 12/2/13 12:57 PM, Joe Stein wrote:> Neha, as far as the release process is this what you had in mind> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process or> different content or more of something or such?>> Per the POM, I was able to use the artifacts from the maven repository> without having to-do anything more than just specifying the artifacts with> sbt.>> resolvers += "Apache Staging" at "> https://repository.apache.org/content/groups/staging/">> libraryDependencies ++= Seq(> ...,> "org.apache.kafka" % "kafka_2.10" % "0.8.0",> ....> )>> and on the pure maven side> <repositories>> <repository>> <id>ApacheStaging</id>> <url>https://repository.apache.org/content/groups/staging/</url>> </repository>> ...> <dependency>> <groupId>org.apache.kafka</groupId>> <artifactId>kafka_2.9.2</artifactId>> <version>0.8.0</version>> <exclusions>> <exclusion>> <groupId>log4j</groupId>> <artifactId>log4j</artifactId>> </exclusion>> </exclusions>> </dependency>>> which very closely mirrors what David was talking about with ivy as well...> I didn't really think much of it just a matter of XML we can document> (there is actually no using maven documentation on the site at all we> should correct that in any case TBD post release) but if folks find it to> be a pain then we should definitely fix it for sure. off the top of my> head I don't see how to-do that in the Build.scala but I really don't> expect it to be too difficult to figure out... the question is do we hold> it off for 0.8.1 since technically nothing is breaking (like the null> pointer exceptions we had for the bonked pom in beta1 that I shipped to> maven central).>> Before canceling the vote can we at least get consensus to what we are> canceling and exactly what fixes should be in RC6 or ... agree to ship RC5> and hold whatever is left for 0.8.1>> I am totally fine with working on RC6 (actually just cancelled my plans for> the evening because of a whole slew of client work that hit my plate) but I> want to make sure we have everything covered that everyone that is voting> expects to be in there.>> David, a few items below don't make sense I sent another email on the> thread in regards to the LICENSE>>> /*******************************************> Joe Stein> Founder, Principal Consultant> Big Data Open Source Security LLC> http://www.stealth.ly> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>> ********************************************/>>> On Mon, Dec 2, 2013 at 12:19 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:>>> I think we should maintain a wiki describing the release process in detail,>> so we save the turnaround time on a release. We can have a VOTE thread to>> agree on the release guidelines and follow it. Having said that, it is>> worth having the correct .pom file at the very least, since the release is>> not very useful if people cannot consume it without pain.>>>> Thanks,>> Neha>>>>>> On Mon, Dec 2, 2013 at 8:59 AM, Joe Stein <[EMAIL PROTECTED]> wrote:>>>>> General future thought comment first: lets be careful please to raising

> Inline:>>> On 12/2/13 11:59 AM, Joe Stein wrote:>>> General future thought comment first: lets be careful please to raising>> issues as show stoppers that have been there previously (especially if>> greater than one version previous release back also has the problem) and>> can get fixed in a subsequent release and is only now more pressing>> because>> we know about them... seeing something should not necessarily always>> create>> priority (sometimes sure, of course but not always that is not the best>> way>> to manage changes). The VOTE thread should be to artifacts and what we>> are>> releasing as proper and correct per Apache guidelines... and to make sure>> that the person doing the release doesn't do something incorrect ... like>> using the wrong version of JDK to build =8^/. If we are not happy with>> release as ready to ship then lets not call a VOTE and save the prolonged>> weeks that drag out with so many release candidates. The community>> suffers>> from this.>>> +1 If we can get most of this release preparation stuff automated, then we> can iterate on it in a release branch before tagging and voting.>> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...>> hopefully a few more hours for other folks to comment and discuss the>> issues you raised with my $0.02852425 included below and follow-ups as>> they>> become necessary... I am also out of pocket in a few hours until tomorrow>> morning so if it passed I would not be able to publish and announce or if>> failed look towards RC6 anyways =8^)>>>> /*******************************************>> Joe Stein>> Founder, Principal Consultant>> Big Data Open Source Security LLC>> http://www.stealth.ly>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>>> ********************************************/>>>>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>>> Seems like most people are verifying the src, so I'll pick on the>>> binaries>>> and Maven stuff ;)>>>>>> A few problems I see:>>>>>> There are some vestigial Git files in the src download: an empty .git and>>> .gitignore>>>>>> Ok, I can do a better job with 0.8.1 but I am not sure this is very>> different than beta1 and not necessarily a show stopper for 0.8.0>> requiring>> another release candidate, is it? I think updating the release docs and>> rmdir .git after the rm -fr and rm .gitignore moving forward makes sense.>>> Agreed, not a show stopper.>>>>>> In the source download, I see the SBT license in LICENSE which seems>>> correct (since we distribute an SBT binary), but in the binary download I>>> see the same license. Don't we need the Scala license (>>> http://www.scala-lang.org/license.html) in the binary distribution?>>>>>> I fixed this already not only in the binary release>> https://issues.apache.org/jira/browse/KAFKA-1131 but also in the JAR>> files>> that are published to Maven>> https://issues.apache.org/jira/browse/KAFKA-1133are you checking from>> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/ because I just>> downloaded again and it looks alright to me. If not then definitely this>> RC should be shot down because it does not do what we are saying it is>> doing.. but if it is wrong can you be more specific and create a JIRA with>> the fix because I thought I got it right already... but if not then lets>> get it right because that is why we pulled the release in RC3>>> The LICENSE file in both the src and binary downloads includes "SBT> LICENSE" at the end. I could be wrong, but I think the src download should> include the SBT licnese and the binary download should include the Scala

-DavidOn 12/3/13 1:41 AM, Jun Rao wrote:> The release voting is based on lazy majority (> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Voting). So> a -1 doesn't kill the release. The question is whether those issues are> really show stoppers.>> Thanks,>> Jun>>>>> On Mon, Dec 2, 2013 at 10:19 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>> Inline:>>>>>> On 12/2/13 11:59 AM, Joe Stein wrote:>>>>> General future thought comment first: lets be careful please to raising>>> issues as show stoppers that have been there previously (especially if>>> greater than one version previous release back also has the problem) and>>> can get fixed in a subsequent release and is only now more pressing>>> because>>> we know about them... seeing something should not necessarily always>>> create>>> priority (sometimes sure, of course but not always that is not the best>>> way>>> to manage changes). The VOTE thread should be to artifacts and what we>>> are>>> releasing as proper and correct per Apache guidelines... and to make sure>>> that the person doing the release doesn't do something incorrect ... like>>> using the wrong version of JDK to build =8^/. If we are not happy with>>> release as ready to ship then lets not call a VOTE and save the prolonged>>> weeks that drag out with so many release candidates. The community>>> suffers>>> from this.>>>>> +1 If we can get most of this release preparation stuff automated, then we>> can iterate on it in a release branch before tagging and voting.>>>> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...>>> hopefully a few more hours for other folks to comment and discuss the>>> issues you raised with my $0.02852425 included below and follow-ups as>>> they>>> become necessary... I am also out of pocket in a few hours until tomorrow>>> morning so if it passed I would not be able to publish and announce or if>>> failed look towards RC6 anyways =8^)>>>>>> /*******************************************>>> Joe Stein>>> Founder, Principal Consultant>>> Big Data Open Source Security LLC>>> http://www.stealth.ly>>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>>>> ********************************************/>>>>>>>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>>>>> Seems like most people are verifying the src, so I'll pick on the>>>> binaries>>>> and Maven stuff ;)>>>>>>>> A few problems I see:>>>>>>>> There are some vestigial Git files in the src download: an empty .git and>>>> .gitignore>>>>>>>> Ok, I can do a better job with 0.8.1 but I am not sure this is very>>> different than beta1 and not necessarily a show stopper for 0.8.0>>> requiring>>> another release candidate, is it? I think updating the release docs and>>> rmdir .git after the rm -fr and rm .gitignore moving forward makes sense.>>>>> Agreed, not a show stopper.>>>>>>> In the source download, I see the SBT license in LICENSE which seems>>>> correct (since we distribute an SBT binary), but in the binary download I>>>> see the same license. Don't we need the Scala license (>>>> http://www.scala-lang.org/license.html) in the binary distribution?>>>>>>>> I fixed this already not only in the binary release>>> https://issues.apache.org/jira/browse/KAFKA-1131 but also in the JAR>>> files>>> that are published to Maven>>> https://issues.apache.org/jira/browse/KAFKA-1133are you checking from>>> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/ because I just>>> downloaded again and it looks alright to me. If not then definitely this>>> RC should be shot down because it does not do what we are saying it is

> Not to get too side tracked, but I think lazy consensus is supposed to> mean "silence gives assent" http://www.apache.org/foundation/voting.html#> LazyConsensus>> After the release, we should clean up the language of the bylaws to match> the language here http://www.apache.org/foundation/glossary.html>> -David>> On 12/3/13 1:41 AM, Jun Rao wrote:>>> The release voting is based on lazy majority (>> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Voting).>> So>> a -1 doesn't kill the release. The question is whether those issues are>> really show stoppers.>>>> Thanks,>>>> Jun>>>>>>>>>> On Mon, Dec 2, 2013 at 10:19 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>>> Inline:>>>>>>>>> On 12/2/13 11:59 AM, Joe Stein wrote:>>>>>> General future thought comment first: lets be careful please to raising>>>> issues as show stoppers that have been there previously (especially if>>>> greater than one version previous release back also has the problem) and>>>> can get fixed in a subsequent release and is only now more pressing>>>> because>>>> we know about them... seeing something should not necessarily always>>>> create>>>> priority (sometimes sure, of course but not always that is not the best>>>> way>>>> to manage changes). The VOTE thread should be to artifacts and what we>>>> are>>>> releasing as proper and correct per Apache guidelines... and to make>>>> sure>>>> that the person doing the release doesn't do something incorrect ...>>>> like>>>> using the wrong version of JDK to build =8^/. If we are not happy with>>>> release as ready to ship then lets not call a VOTE and save the>>>> prolonged>>>> weeks that drag out with so many release candidates. The community>>>> suffers>>>> from this.>>>>>>>> +1 If we can get most of this release preparation stuff automated,>>> then we>>> can iterate on it in a release branch before tagging and voting.>>>>>> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...>>>>>>> hopefully a few more hours for other folks to comment and discuss the>>>> issues you raised with my $0.02852425 included below and follow-ups as>>>> they>>>> become necessary... I am also out of pocket in a few hours until>>>> tomorrow>>>> morning so if it passed I would not be able to publish and announce or>>>> if>>>> failed look towards RC6 anyways =8^)>>>>>>>> /*******************************************>>>> Joe Stein>>>> Founder, Principal Consultant>>>> Big Data Open Source Security LLC>>>> http://www.stealth.ly>>>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>>>>> ********************************************/>>>>>>>>>>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]> wrote:>>>>>>>> Seems like most people are verifying the src, so I'll pick on the>>>>>>>>> binaries>>>>> and Maven stuff ;)>>>>>>>>>> A few problems I see:>>>>>>>>>> There are some vestigial Git files in the src download: an empty .git>>>>> and>>>>> .gitignore>>>>>>>>>> Ok, I can do a better job with 0.8.1 but I am not sure this is very>>>>>>>>> different than beta1 and not necessarily a show stopper for 0.8.0>>>> requiring>>>> another release candidate, is it? I think updating the release docs and>>>> rmdir .git after the rm -fr and rm .gitignore moving forward makes>>>> sense.>>>>>>>> Agreed, not a show stopper.>>>>>>>>> In the source download, I see the SBT license in LICENSE which seems>>>>>>>>> correct (since we distribute an SBT binary), but in the binary>>>>> download I>>>>> see the same license. Don't we need the Scala license (>>>>> http://www.scala-lang.org/license.html) in the binary distribution?>>>>>

> There is a difference btw lazy consensus and lazy majority. See>> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvalsfor> the precise definition.>> Thanks,>> Jun>>> On Tue, Dec 3, 2013 at 6:06 AM, David Arthur <[EMAIL PROTECTED]> wrote:>> > Not to get too side tracked, but I think lazy consensus is supposed to> > mean "silence gives assent"> http://www.apache.org/foundation/voting.html#> > LazyConsensus> >> > After the release, we should clean up the language of the bylaws to match> > the language here http://www.apache.org/foundation/glossary.html> >> > -David> >> > On 12/3/13 1:41 AM, Jun Rao wrote:> >> >> The release voting is based on lazy majority (> >> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Voting> ).> >> So> >> a -1 doesn't kill the release. The question is whether those issues are> >> really show stoppers.> >>> >> Thanks,> >>> >> Jun> >>> >>> >>> >>> >> On Mon, Dec 2, 2013 at 10:19 AM, David Arthur <[EMAIL PROTECTED]> wrote:> >>> >> Inline:> >>>> >>>> >>> On 12/2/13 11:59 AM, Joe Stein wrote:> >>>> >>> General future thought comment first: lets be careful please to> raising> >>>> issues as show stoppers that have been there previously (especially if> >>>> greater than one version previous release back also has the problem)> and> >>>> can get fixed in a subsequent release and is only now more pressing> >>>> because> >>>> we know about them... seeing something should not necessarily always> >>>> create> >>>> priority (sometimes sure, of course but not always that is not the> best> >>>> way> >>>> to manage changes). The VOTE thread should be to artifacts and what> we> >>>> are> >>>> releasing as proper and correct per Apache guidelines... and to make> >>>> sure> >>>> that the person doing the release doesn't do something incorrect ...> >>>> like> >>>> using the wrong version of JDK to build =8^/. If we are not happy> with> >>>> release as ready to ship then lets not call a VOTE and save the> >>>> prolonged> >>>> weeks that drag out with so many release candidates. The community> >>>> suffers> >>>> from this.> >>>>> >>>> +1 If we can get most of this release preparation stuff automated,> >>> then we> >>> can iterate on it in a release branch before tagging and voting.> >>>> >>> ok, now on to RC5 ...lets extend the vote until 12pm PT tomorrow ...> >>>> >>>> hopefully a few more hours for other folks to comment and discuss the> >>>> issues you raised with my $0.02852425 included below and follow-ups as> >>>> they> >>>> become necessary... I am also out of pocket in a few hours until> >>>> tomorrow> >>>> morning so if it passed I would not be able to publish and announce or> >>>> if> >>>> failed look towards RC6 anyways =8^)> >>>>> >>>> /*******************************************> >>>> Joe Stein> >>>> Founder, Principal Consultant> >>>> Big Data Open Source Security LLC> >>>> http://www.stealth.ly> >>>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>> >>>> ********************************************/> >>>>> >>>>> >>>> On Mon, Dec 2, 2013 at 11:00 AM, David Arthur <[EMAIL PROTECTED]>> wrote:> >>>>> >>>> Seems like most people are verifying the src, so I'll pick on the> >>>>> >>>>> binaries> >>>>> and Maven stuff ;)> >>>>>> >>>>> A few problems I see

> Talking about binary release, do we really need to include bin/run-rat.sh?> As far as I understand it is only used to bring licenses to the scene and> quite redundant for already baked release.>> Next, I not quite sure, but probably it makes sense to drop> libs/scala-compiler.jar -- kafka do not perform compilations during runtime> and this step will trim some fat from the resulting release (from 17 mb> down to 9.5 mb*).>> I managed to satisfy maven with only two exclusions, but yes, it would be> good to see them in original pom.>> * by the way using the best possible compression method (-9 instead of> default -6) + drop of compiler lib gave me the very same result -- 9.5 Mb>>> 2013/12/2 David Arthur <[EMAIL PROTECTED]>>> > Seems like most people are verifying the src, so I'll pick on the> binaries> > and Maven stuff ;)> >> > A few problems I see:> >> > There are some vestigial Git files in the src download: an empty .git and> > .gitignore> >> > In the source download, I see the SBT license in LICENSE which seems> > correct (since we distribute an SBT binary), but in the binary download I> > see the same license. Don't we need the Scala license (> > http://www.scala-lang.org/license.html) in the binary distribution?> >> > I create a simple Ant+Ivy project to test resolving the artifacts> > published to Apache staging repo: https://github.com/mumrah/kafka-ivy.> > This will fetch Kafka libs from the Apache staging area and other things> > from Maven Central. It will fetch the jars into lib/ivy/{conf} and> generate> > a report of the dependencies, conflicts, and licenses into ivy-report.> > Notice I had to add three exclusions to get things working. Maybe we> should> > add these to our pom?> >> > I think I'll have to -1 the release due to the missing Scala license in> > the binary dist. We should check the other licenses as well (see> ivy-report> > from my little Ant project).> >> > -David> >> > On 11/26/13 5:34 PM, Joe Stein wrote:> >> >> This is the fifth candidate for release of Apache Kafka 0.8.0. This> >> release candidate is now built from JDK 6 as RC4 was built with JDK 7.> >>> >> Release Notes for the 0.8.0 release> >> http://people.apache.org/~joestein/kafka-0.8.0-> >> candidate5/RELEASE_NOTES.html> >>> >> *** Please download, test and vote by Monday December, 2nd, 12pm PDT> >>> >> Kafka's KEYS file containing PGP keys we use to sign the release:> >> http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5 and> >> sha1> >> checksum> >>> >> * Release artifacts to be voted upon (source and binary):> >> http://people.apache.org/~joestein/kafka-0.8.0-candidate5/> >>> >> * Maven artifacts to be voted upon prior to release:> >> https://repository.apache.org/content/groups/staging/> >>> >> (i.e. in sbt land this can be added to the build.sbt to use Kafka> >> resolvers += "Apache Staging" at "> >> https://repository.apache.org/content/groups/staging/"> >> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"> >> )> >>> >> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag> >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=> >> 2c20a71a010659e25af075a024cbd692c87d4c89> >>> >> /*******************************************> >> Joe Stein

> Joe, another thing I noticed in the staging repo:>> The POM for 2.8.2, 2.9.1, 2.9.2, and 2.10 include scala-compiler and some> other stuff that is not included in 2.8.0>> 2.8.0 https://repository.apache.org/content/groups/staging/org/> apache/kafka/kafka_2.8.0/0.8.0/kafka_2.8.0-0.8.0.pom> 2.8.2 https://repository.apache.org/content/groups/staging/org/> apache/kafka/kafka_2.8.2/0.8.0/kafka_2.8.2-0.8.0.pom>> Here's a diff of those two: https://gist.github.com/> mumrah/7bd6bd8e2805210d5d9d/revisions>> I think maybe the 2.8.0 POM is missing some stuff it needs (zkclient,> snappy, yammer metrics). And there is a duplicate ZK entry for the POMs> >2.8.0>> -David>>>>> On 12/2/13 12:57 PM, Joe Stein wrote:>>> Neha, as far as the release process is this what you had in mind>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process or>> different content or more of something or such?>>>> Per the POM, I was able to use the artifacts from the maven repository>> without having to-do anything more than just specifying the artifacts with>> sbt.>>>> resolvers += "Apache Staging" at ">> https://repository.apache.org/content/groups/staging/">>>> libraryDependencies ++= Seq(>> ...,>> "org.apache.kafka" % "kafka_2.10" % "0.8.0",>> ....>> )>>>> and on the pure maven side>> <repositories>>> <repository>>> <id>ApacheStaging</id>>> <url>https://repository.apache.org/content/groups/staging/>> </url>>> </repository>>> ...>> <dependency>>> <groupId>org.apache.kafka</groupId>>> <artifactId>kafka_2.9.2</artifactId>>> <version>0.8.0</version>>> <exclusions>>> <exclusion>>> <groupId>log4j</groupId>>> <artifactId>log4j</artifactId>>> </exclusion>>> </exclusions>>> </dependency>>>>> which very closely mirrors what David was talking about with ivy as>> well...>> I didn't really think much of it just a matter of XML we can document>> (there is actually no using maven documentation on the site at all we>> should correct that in any case TBD post release) but if folks find it to>> be a pain then we should definitely fix it for sure. off the top of my>> head I don't see how to-do that in the Build.scala but I really don't>> expect it to be too difficult to figure out... the question is do we hold>> it off for 0.8.1 since technically nothing is breaking (like the null>> pointer exceptions we had for the bonked pom in beta1 that I shipped to>> maven central).>>>> Before canceling the vote can we at least get consensus to what we are>> canceling and exactly what fixes should be in RC6 or ... agree to ship RC5>> and hold whatever is left for 0.8.1>>>> I am totally fine with working on RC6 (actually just cancelled my plans>> for>> the evening because of a whole slew of client work that hit my plate) but>> I>> want to make sure we have everything covered that everyone that is voting>> expects to be in there.>>>> David, a few items below don't make sense I sent another email on the>> thread in regards to the LICENSE>>>>>> /*******************************************>> Joe Stein>> Founder, Principal Consultant>> Big Data Open Source Security LLC>> http://www.stealth.ly>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>