Alex Gray
added a comment - 30/Apr/13 02:39 Thanks Scott, for the links. Just searching for "sbt s3" does not get too many hits, but the best one by far is:
http://stackoverflow.com/questions/9958103/how-to-publish-to-amazon-s3-with-sbt
So far, from what I can tell, either copy the ivy jar and do the following (the million dollar question is how to configure the setters):
{
val s3r = new org.springframework.aws.ivy.S3Resolver
// call its setters to configure it, see api docs above etc.
publishTo := Some(new sbt.RawRepository(s3r)
}
or use sbt-s3, which I haven't had any success in.
Nonetheless, I think I have enough to experiment with.
Thanks!

The build is using SBT, whic is using Ivy behind the scenes, so there is no pom.xml. One is generated during the build and can be published to a maven repo if SBT is configured correctly. I'm not familiar with how to do that.

Scott Carey
added a comment - 30/Apr/13 00:46 The build is using SBT, whic is using Ivy behind the scenes, so there is no pom.xml. One is generated during the build and can be published to a maven repo if SBT is configured correctly. I'm not familiar with how to do that.
The best I can find quickly with a search are:
https://groups.google.com/forum/?fromgroups=#!topic/simple-build-tool/CJS8GvXO4j0
https://groups.google.com/forum/?fromgroups=#!topic/simple-build-tool/HaEv4P9Mxkw

Silly question, but I just downloaded 0.8 branch and I see no pom.xml.
I would like to locally build kafka and publish to our private local maven repo.
Is this currently possible? Do I have to do something for the pom.xml to get generated.
Sorry for the newbie question.

I noticed that when I run ./sbt publish-local, the artifacts get pushed to my local repo:
~/.ivy2/local/org.apache/contrib_2.8.0
~/.ivy2/local/org.apache/kafka_2.8.0

I would like to publish to our private local maven repo (which just so happens to be in S3), but I'm used to modifying a pom.xml to add the required plugins/repoID's/etc/etc/, hence the question on where the pom.xml is.

Alex Gray
added a comment - 29/Apr/13 19:39 Silly question, but I just downloaded 0.8 branch and I see no pom.xml.
I would like to locally build kafka and publish to our private local maven repo.
Is this currently possible? Do I have to do something for the pom.xml to get generated.
Sorry for the newbie question.
I noticed that when I run ./sbt publish-local, the artifacts get pushed to my local repo:
~/.ivy2/local/org.apache/contrib_2.8.0
~/.ivy2/local/org.apache/kafka_2.8.0
I would like to publish to our private local maven repo (which just so happens to be in S3), but I'm used to modifying a pom.xml to add the required plugins/repoID's/etc/etc/, hence the question on where the pom.xml is.
Thanks!

Darren Sargent
added a comment - 25/Mar/13 20:42 This ticket is marked as fixed, but the latest on the 0.8 branch still depends on an unpublished version of Yammer - this is from Build.scala:
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>3.0.0-c0c8be71</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-annotations</artifactId>
<version>3.0.0-c0c8be71</version>
<scope>compile</scope>
</dependency>
Maven Central doesn't have a version anything like the one mentioned above: http://repo2.maven.org/maven2/com/yammer/metrics/metrics-core/

Additionally, 0.8.x can not be an official Apache release in this form because the source tarball would not compile or provide all of the tools necessary to build or reference publicly available libraries (i.e. the metrics snapshot jar is ephemeral).

Scott Carey
added a comment - 23/Jan/13 21:30 Additionally, 0.8.x can not be an official Apache release in this form because the source tarball would not compile or provide all of the tools necessary to build or reference publicly available libraries (i.e. the metrics snapshot jar is ephemeral).

I currently use yammer-metrics in my code, and was excited to see that kafka was going to use metrics so that we could clean up and standardize things there. However, since it depends essentially on trunk (3.0.0-SNAPSHOT) including Kafka 0.8 breaks and there is no release plan or date for when 3.0.0 will even exist.

Even worse, if we try and compile against the CURRENT yammer-metrics head 3.0.0-SNAPSHOT kafka fails to compile because it is ... trunk and things change.

Scott Carey
added a comment - 23/Jan/13 21:26 0.8.x needs to depend on a released version of metrics.
I currently use yammer-metrics in my code, and was excited to see that kafka was going to use metrics so that we could clean up and standardize things there. However, since it depends essentially on trunk (3.0.0-SNAPSHOT) including Kafka 0.8 breaks and there is no release plan or date for when 3.0.0 will even exist.
Even worse, if we try and compile against the CURRENT yammer-metrics head 3.0.0-SNAPSHOT kafka fails to compile because it is ... trunk and things change.
0.8.x needs to depend on a released version of metrics.

Are we expecting 0.7.3 release with feature to publish to maven repository..? If so do you know the time frame.. ? Also publish-local publish it to local ivy repository.. Is there a way to change it to publish to local maven repository..?

Rajasekar Elango
added a comment - 22/Jan/13 21:12 - edited Are we expecting 0.7.3 release with feature to publish to maven repository..? If so do you know the time frame.. ? Also publish-local publish it to local ivy repository.. Is there a way to change it to publish to local maven repository..?
Raja.

I'm new to maven, so this may be a dumb question: would it be reasonable/easy to publish nightly jars (via Apache?), under the 0.8.0-SNAPSHOT tag? I have an automated build system that currently git's and builds the whole thing, which I would love to replace with simple jar files.

ben fleis
added a comment - 10/Jan/13 10:43 I'm new to maven, so this may be a dumb question: would it be reasonable/easy to publish nightly jars (via Apache?), under the 0.8.0-SNAPSHOT tag? I have an automated build system that currently git's and builds the whole thing, which I would love to replace with simple jar files.

Joe Stein
added a comment - 10/Jan/13 00:31 << Just to clarify – this is marked as resolved, but I don't see any artifacts in maven central (search.maven.org). Is there some other repo to which the artifacts have been released?
0.8 is not released yet. once it is the artifacts for 0.8 will be published

Evan Pollan
added a comment - 09/Jan/13 22:01 Just to clarify – this is marked as resolved, but I don't see any artifacts in maven central (search.maven.org). Is there some other repo to which the artifacts have been released?

1) removed the commented out line of code that is not needed anymore
2) overrode the organization to be org.apache so the groupId shows up as such

sbt works for using this locally now

"org.apache" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT"

it is a great step forward toward for helping folks to use the core code. thanks!

any other issues that come up I will follow-up accordingly with them and will shepherd that we publish once released (since we are not there yet it is premature but saw no reason to not get these changes up in trunk and resolve this ticket)

Joe Stein
added a comment - 07/Jan/13 03:51 I committed this with 2 minor changes
1) removed the commented out line of code that is not needed anymore
2) overrode the organization to be org.apache so the groupId shows up as such
sbt works for using this locally now
"org.apache" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT"
it is a great step forward toward for helping folks to use the core code. thanks!
any other issues that come up I will follow-up accordingly with them and will shepherd that we publish once released (since we are not there yet it is premature but saw no reason to not get these changes up in trunk and resolve this ticket)

this looks really good so far I was able to get ./sbt publish-local to run just fine and integrate the result loca .iv2 with another sbt project I have using "kafka" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT"

couple things

1) the groupID should be = org.apache
2) do you see any reason this would not run on the 0.7.2 branch? I am going to propose a 0.7.3 release with this change also in there (less the metrics as that is only in 0.8 and changes to version all minor details). I don't but figure I would ask

Joe Stein
added a comment - 06/Jan/13 06:45 this looks really good so far I was able to get ./sbt publish-local to run just fine and integrate the result loca .iv2 with another sbt project I have using "kafka" % "core-kafka_2.8.0" % "0.8.0-SNAPSHOT"
couple things
1) the groupID should be = org.apache
2) do you see any reason this would not run on the 0.7.2 branch? I am going to propose a 0.7.3 release with this change also in there (less the metrics as that is only in 0.8 and changes to version all minor details). I don't but figure I would ask
notes for self: http://www.apache.org/dev/publishing-maven-artifacts.html

Maxime Brugidou
added a comment - 04/Jan/13 09:13 Taking Otis notes into account, Here is the simplest patch I could make, that applies cleanly on 0.8:
Here are the changes:
Version is 0.8.0-SNAPSHOT until 0.8.0 becomes stable
You need to change the publish URL and credentials location
Use com.101tec.zkclient 0.2
Use com.yammer.metrics.metrics-* 3.0.0-SNAPSHOT (I had to use snapshot because there is no 3.0.0 stable)
Use org.slf4j.slf4j-simple 1.6.4 ("latest.release" was used)
To publish:
./sbt publish-local && ./sbt publish
This will create a pom for kafka that depends on all the subprojects. The "main" project that you generally want to use is core-kafka_2.8.0 like this:
<dependency>
<groupId>kafka</groupId>
<artifactId>core-kafka_2.8.0</artifactId>
<version>0.8.0-SNAPSHOT</version>
</dependency>

Otis Gospodnetic
added a comment - 29/Nov/12 21:26 There you have it:
zkclient 0.2:
http://search.maven.org/#artifactdetails%7Ccom.101tec%7Czkclient%7C0.2%7Cjar
metrics 2.2.0:
http://search.maven.org/#artifactdetails%7Ccom.yammer.metrics%7Cmetrics-parent%7C2.2.0%7Cpom
This is now the most popular Kafka issue by far!

Otis Gospodnetic
added a comment - 27/Nov/12 04:31 Update on the zklient front - it's getting Sonatyped: https://issues.sonatype.org/browse/OSSRH-4783 and Johannes Zillmann says it should be in there in 2-3 days.

2. Second thing is to create a ticket at https://issues.sonatype.org/browse/OSSRH with 'Project' type and
subtype of 'Support - Open Source Project Repository Hosting' and
issue type of 'New Project'. You need to fill all the information
there and submit issue. After that you'll have to wait from a few
hours to two days for it to be ready.

4. Add GPG maven plugin to your project. The best way is to create a
new profile, not to sign artifacts during local builds. With an
profile like the one below, you use if by adding -DperformRelease=true
to maven command. It will ask for your key password and sign artifacts
with your key. This is needed to publish artifacts to maven
repository. I've created the following profile:

10. After that, your artifcts should be uloaded to staging sonatype.
You now should login into https://oss.sonatype.org/ with the
previously created account, choose repository and click "Close". If
you did everything right, close will be successful.

11. Now click "Release" on the closed repository and comment on the
issue created in step 2) that you released the artifacts. Someone at
Sonatype will review the artifacts and accept it. The review process
will only happen once and the initial release. Later releases will be
released automaticaly.

Otis Gospodnetic
added a comment - 20/Nov/12 03:47 Here is the procedure that one of Sematext engineers shared with the rest of our team when he set up one of our OSS projects in Sonatype's Maven repo (so ignore references to ActionGenerator):
1. The first thing is to create an account at
https://issues.sonatype.org/ .
2. Second thing is to create a ticket at
https://issues.sonatype.org/browse/OSSRH with 'Project' type and
subtype of 'Support - Open Source Project Repository Hosting' and
issue type of 'New Project'. You need to fill all the information
there and submit issue. After that you'll have to wait from a few
hours to two days for it to be ready.
3. After 2) is ready you need to install GPG client, create your key
and publish it:
gpg --gen-key (to generate key)
gpg --list-keys (to list keys)
gpg --keyserver hkp://pool.sks-keyservers.net --send-keys YOUR_KEY_ID (to upload it)
gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys YOUR_KEY_ID (to check if it was uploaded)
4. Add GPG maven plugin to your project. The best way is to create a
new profile, not to sign artifacts during local builds. With an
profile like the one below, you use if by adding -DperformRelease=true
to maven command. It will ask for your key password and sign artifacts
with your key. This is needed to publish artifacts to maven
repository. I've created the following profile:
<profiles>
<profile>
<id>release-sign-artifacts</id>
<activation>
<property>
<name>performRelease</name>
<value>true</value>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.1</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
5. You need to ensure that you have the following in your POM files:
<modelVersion>
<groupId>
<artifactId>
<version>
<packaging>
<name>
<description>
<url>
<licenses>
<scm><url>
<scm><connection>
<developers>
6. Add Sonatype parent to your parent POM:
<parent>
<groupId>org.sonatype.oss</groupId>
<artifactId>oss-parent</artifactId>
<version>7</version>
</parent>
7. Add SCM information to your project. For example like this:
<scm>
<connection>scm:git:git@github.com:sematext/ActionGenerator.git</connection>
<developerConnection>scm:git:git@github.com:sematext/ActionGenerator.git</developerConnection>
<url> https://github.com/sematext/ActionGenerator </url>
</scm>
8. Modify your settings.xml file (you should have it in your maven
home directory) and add the following:
<servers>
<server>
<id>sonatype-nexus-snapshots</id>
<username>your-jira-id</username>
<password>your-jira-pwd</password>
</server>
<server>
<id>sonatype-nexus-staging</id>
<username>your-jira-id</username>
<password>your-jira-pwd</password>
</server>
</servers>
Where:
your-jira-id is the ID of your account created in step 1
your-jira-pwd is the password for account created in step 1
9. Build your project, sign artifacts and deploy:
$mvn clean install deploy -DperformRelease=true
10. After that, your artifcts should be uloaded to staging sonatype.
You now should login into https://oss.sonatype.org/ with the
previously created account, choose repository and click "Close". If
you did everything right, close will be successful.
11. Now click "Release" on the closed repository and comment on the
issue created in step 2) that you released the artifacts. Someone at
Sonatype will review the artifacts and accept it. The review process
will only happen once and the initial release. Later releases will be
released automaticaly.
This is what I did. Now I'm waiting for ActionGenerator to be
released. In case you would like to go deeper into that process, there
is a nice documentation available at
https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8?
Just changed the affected version to 0.8

Nobody is assigned. Is it known who will do this?
Anyone who is interested can pick this up. Feel free to assign this to yourself.

It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?KAFKA-134 is required only to be able to publish to maven using an sbt command. For 0.8, if that is not ready, we can manually publish it. All the assignee would do in this case, is figure out the procedure to get this done and post it here. One of the committers can help roll the ball.

Neha Narkhede
added a comment - 18/Nov/12 22:40
How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8?
Just changed the affected version to 0.8
Nobody is assigned. Is it known who will do this?
Anyone who is interested can pick this up. Feel free to assign this to yourself.
It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?
KAFKA-134 is required only to be able to publish to maven using an sbt command. For 0.8, if that is not ready, we can manually publish it. All the assignee would do in this case, is figure out the procedure to get this done and post it here. One of the committers can help roll the ball.

FYI just emailed Stefan asking if zkclient 0.2 could be cut and published in Maven. He replied a few minutes later from a cab and CCed his colleague asking him to do this, so if stars align, you may see zkclient 0.2 out and you can remove it from Kafka.

And then I really really hope you can publish Kafka to a Maven repo!

Questions/comments:

How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8?

Nobody is assigned. Is it known who will do this?

It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?

Otis Gospodnetic
added a comment - 16/Nov/12 04:21 FYI just emailed Stefan asking if zkclient 0.2 could be cut and published in Maven. He replied a few minutes later from a cab and CCed his colleague asking him to do this, so if stars align, you may see zkclient 0.2 out and you can remove it from Kafka.
And then I really really hope you can publish Kafka to a Maven repo!
Questions/comments:
How come this is not Fix Version 0.8? Should it not be? Is it not planned for the upcoming 0.8?
Nobody is assigned. Is it known who will do this?
It looks like KAFKA-134 is marked as blocker, but it's also not assigned and is not set for 0.8...?

Neha Narkhede
added a comment - 14/Nov/12 05:07 >> So is the blocker for this issue the fact that there is a custom zkclient?
I believe so.
>> Should the modified API be provided as a patch to the original zkclient?
It is already checked into zkclient trunk
>> Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo?
Sure, please feel free to ping them.

Thanks Neha.
So is the blocker for this issue the fact that there is a custom zkclient?
Should the modified API be provided as a patch to the original zkclient?
Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo?

Otis Gospodnetic
added a comment - 14/Nov/12 04:13 Thanks Neha.
So is the blocker for this issue the fact that there is a custom zkclient?
Should the modified API be provided as a patch to the original zkclient?
Should the zkclient maintainers be nudged to publish the new zkclient to a Maven repo?

Neha Narkhede
added a comment - 13/Nov/12 16:56 >> Custom zkclient jar... maybe a Kafka developer in the know can jump in?
That's because we modified one of the zk client APIs to expose the stat object returned from the underlying zookeeper APIs. But after that, zkclient was never published to Maven.

Otis Gospodnetic
added a comment - 13/Nov/12 00:09 I noticed Fix Version is not set, but if you look at https://issues.apache.org/jira/browse/KAFKA#selectedTab=com.atlassian.jira.plugin.system.project%3Apopularissues-panel you'll see this is THE most popular Kafka issue.
Shouldn't this be set to 0.8, so it gets done for/with the upcoming release?

Custom zkclient jar... maybe a Kafka developer in the know can jump in?
Maybe this jar could be included/hidden in one of the Kafka jars, so it is not treated as external dependencies?
Oh, is that zkclient Stefan's zkclient? This one: http://mvnrepository.com/artifact/com.github.sgroschupf/zkclient/0.1 but locally modified, jarred, and stuck in svn?
Maybe we could instead provide a patch for zkclient and get zkclient maintainer(s) to apply it? I knows Stefan personally and could bug him about it.... he is a busy CEO these days, but maybe somebody else on his team could get this done.

Otis Gospodnetic
added a comment - 13/Nov/12 00:07 - edited Coda Hale metrics does seem to be in Maven repo already, see: http://mvnrepository.com/artifact/com.yammer.metrics
Though from KAFKA-585 it looks like there may be a dependency on Metrics 3.*, which is not in any Maven repo yet.
Custom zkclient jar... maybe a Kafka developer in the know can jump in?
Maybe this jar could be included/hidden in one of the Kafka jars, so it is not treated as external dependencies?
Oh, is that zkclient Stefan's zkclient? This one: http://mvnrepository.com/artifact/com.github.sgroschupf/zkclient/0.1 but locally modified, jarred, and stuck in svn?
Maybe we could instead provide a patch for zkclient and get zkclient maintainer(s) to apply it? I knows Stefan personally and could bug him about it.... he is a busy CEO these days, but maybe somebody else on his team could get this done.

Maybe the thing to do then is to help owners of those projects Kafka depends on get their stuff into Maven, starting by opening tickets in their issue tracker.Sam Meder or Sam Meder - which dependencies are those specifically?

Otis Gospodnetic
added a comment - 12/Nov/12 16:37 Maybe the thing to do then is to help owners of those projects Kafka depends on get their stuff into Maven, starting by opening tickets in their issue tracker.
Sam Meder or Sam Meder - which dependencies are those specifically?

+1 for this. Funny, it looks like it has been exactly 365 days since the issue was opened and a bit over 6 months since the last comment. KAFKA-134 is still blocking... Jay Kreps, Joe Stein, & co, any chance of committing KAFKA-134 to unblock this issue (9 votes)?

Otis Gospodnetic
added a comment - 15/Sep/12 05:32 +1 for this. Funny, it looks like it has been exactly 365 days since the issue was opened and a bit over 6 months since the last comment. KAFKA-134 is still blocking... Jay Kreps , Joe Stein , & co, any chance of committing KAFKA-134 to unblock this issue (9 votes)?

The pom.xml contained in this gist, coupled with the artifact built by the source contents of the kafka-0.7-incubating-src archive can be deployed to a maven repository and ease things for java clients.

Pierre-Yves Ritschard
added a comment - 10/Jan/12 17:33 The pom.xml contained in this gist, coupled with the artifact built by the source contents of the kafka-0.7-incubating-src archive can be deployed to a maven repository and ease things for java clients.

I came into Mountain View today on a loooong train ride and my internet died right after I posted this. I wanted to say that it wasn't quite ready, but that we should sit down today and go through this so we can perfect it for kafka's long term needs.

Chris Riccomini reminded me about this on Friday and I promised myself I would finish it over the weekend - I think I'm about 3 weeks overdue.

Joshua Hartman
added a comment - 24/Oct/11 20:21 I came into Mountain View today on a loooong train ride and my internet died right after I posted this. I wanted to say that it wasn't quite ready, but that we should sit down today and go through this so we can perfect it for kafka's long term needs.
Chris Riccomini reminded me about this on Friday and I promised myself I would finish it over the weekend - I think I'm about 3 weeks overdue.

Neha Narkhede
added a comment - 24/Oct/11 19:57 Thanks for the patch Josh ! Well in time for 0.7 release.
Please could you attach the diff file here and also select the "grant it to Apache" option ? That way we can apply, test and accept the patch.

Alan Cabrera
added a comment - 15/Sep/11 19:59 - edited I need to state the obvious, because I am a mentor, but while it's good to get this machinery working, this project cannot actually publish any artifacts until its first official release.

Joshua Hartman
added a comment - 15/Sep/11 19:45 I did this last night for one of our other projects, Norbert.
If I upgrade Kafka to sbt 0.10.1, I should be able to do this. Will also write a blog post so other projects know how to do this with maven.