Robert Muir
added a comment - 27/Mar/12 23:19 prototype patch for the lucene core.
You dont need to do anything special for this patch to work: 'ant test' etc works as usual (though only for lucene-core!!!)
i nuked all the libs in lucene/lib/ (junit, ant, ant-junit).
These are instead dependencies of the test-framework.
I have no idea what uses this maven-ant-tasks.jar thats still there, or where to fetch it from, but maybe Steve knows

I have no idea what uses this maven-ant-tasks.jar thats still there, or where to fetch it from, but maybe Steve knows

maven-ant-tasks.jar is used by generate-maven-artifacts / dist-maven / m2-deploy* to deploy Maven artifacts; these are used by the Jenkins Maven builds to deploy snapshots to the Apache Snapshot repository, and up through the most recent release, to create Maven release artifacts to be published on Maven central.

Steve Rowe
added a comment - 27/Mar/12 23:32 I have no idea what uses this maven-ant-tasks.jar thats still there, or where to fetch it from, but maybe Steve knows
maven-ant-tasks.jar is used by generate-maven-artifacts / dist-maven / m2-deploy* to deploy Maven artifacts; these are used by the Jenkins Maven builds to deploy snapshots to the Apache Snapshot repository, and up through the most recent release, to create Maven release artifacts to be published on Maven central.
I unpacked the .jar and found a POM under META-INF, and it says its groupId:artifactId:version coordinates are org.apache.maven:maven-ant-tasks:2.1.3 , which is available from the Maven Central repository at http://repo1.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/ .

Robert Muir
added a comment - 27/Mar/12 23:41
so you could simply fetch it with ivy and make it available to ant using <ivy:cachepatch classpathref="foobar"/> and then load the targets from the classpath in ref="foobar".
I'm not gonna do that now: thats an optimization. Various ant tasks expect libs to be in lib/, so I want to populate lib/, not rewrite the build system.

core+contrib+modules work. though we need to remove jars and move to dependencies.

i found the source of the maven-ant-tasks OOM that hits us in prepare-release. It hit me here too. The problem is that you should not allow the taskdef to run in dist-maven if it already ran. so the one for ivy has unless=ivy.uptodate (which sits in that uptodateAndCompiled propset). Finally i ensured the modules 'crawl' passes this around. We should fix dist-maven the same way.

bogusly every contrib/module currently needs an ivy.xml (even with no dependencies). I'd like to remove this but i just havent mucked with it yet.

Robert Muir
added a comment - 28/Mar/12 00:38 updated patch:
core+contrib+modules work. though we need to remove jars and move to dependencies.
i found the source of the maven-ant-tasks OOM that hits us in prepare-release. It hit me here too. The problem is that you should not allow the taskdef to run in dist-maven if it already ran. so the one for ivy has unless=ivy.uptodate (which sits in that uptodateAndCompiled propset). Finally i ensured the modules 'crawl' passes this around. We should fix dist-maven the same way.
bogusly every contrib/module currently needs an ivy.xml (even with no dependencies). I'd like to remove this but i just havent mucked with it yet.

Great stuff Robert. I've come into the discussion a little late but I'll help where I can.

bogusly every contrib/module currently needs an ivy.xml (even with no dependencies). I'd like to remove this but i just havent mucked with it yet.

I don't think this is so bad? It shows that the contrib/module is properly configured and that the ivy integration was 'done' when the module was created. I think it might also help IDEs if every module is consistent.

Chris Male
added a comment - 28/Mar/12 01:12 Great stuff Robert. I've come into the discussion a little late but I'll help where I can.
bogusly every contrib/module currently needs an ivy.xml (even with no dependencies). I'd like to remove this but i just havent mucked with it yet.
I don't think this is so bad? It shows that the contrib/module is properly configured and that the ivy integration was 'done' when the module was created. I think it might also help IDEs if every module is consistent.

Chris Male
added a comment - 28/Mar/12 12:22 Patch for addressing Solr's example/lib jars.
Adds a build.xml and ivy.xml to solr/example
I haven't committed this since I'm unsure its the best way forward. I've explored other options with ivy (through the use of confs for example) and there doesn't seem to be any other clean way.

Robert Muir
added a comment - 28/Mar/12 13:44 Lets defer any unnecessary refactoring (even if it has benefits) to other issues.
I think we want the minimal (reasonable) patch here that works as close as possible
to the ant build: given the incubator discussion about jars, unfortunately I think
we need to backport this to 3.6.
So its important to keep everything as close as possible to what it was before so that
packaging tasks, etc work.
Seeing how things got working for solr I think we may want to revert even the test-framework/lib
and go back to lucene/lib intentionally just for this purpose: I'll look into this.

Now we have the problematic patched jars and the renamed unreleased jars.

I will start with the patched ones: these are the worst case.
I will look to see if we can have our patch file sitting in svn,
we download the sources (not jar), patch with the patch file,
and compile the thing up.

Robert Muir
added a comment - 28/Mar/12 14:04 Now we have the problematic patched jars and the renamed unreleased jars.
I will start with the patched ones: these are the worst case.
I will look to see if we can have our patch file sitting in svn,
we download the sources (not jar), patch with the patch file,
and compile the thing up.

Hoss Man
added a comment - 29/Mar/12 02:59 I did a touch /home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-LICENSE-FAKE.txt to work around the license checker and this time i wound up with an OOM...
hossman@bester:~/lucene/branch_lucene3930$ ant clean test
....
common.compile-test:
[mkdir] Created dir: /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/sandbox/classes/test
[javac] Compiling 6 source files to /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/sandbox/classes/test
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[copy] Copied 1 empty directory to 1 empty directory under /home/hossman/lucene/branch_lucene3930/lucene/build/contrib/sandbox/classes/test
test-contrib:
[echo] Building demo...
download-ivy:
install-ivy:
resolve:
[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: url = jar:file:/home/hossman/lucene/branch_lucene3930/lucene/ivy/ivy-2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
at java.lang.Throwable.getStackTraceElement(Native Method)
at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
at java.lang.Throwable.printStackTrace(Throwable.java:462)
at java.lang.Throwable.printStackTrace(Throwable.java:451)
at org.apache.tools.ant.Main.startAnt(Main.java:230)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
running with "-verbose" to see if i can get more details on exactly where/why the OOM is happening

Attaching full stderr/stdout of a (differnet "ant -verbose clean test" run that ended with another PermGen OOM...

hossman@bester:~/lucene/branch_lucene3930$ ant -verbose clean test 2>&1 > ant_-verbose_clean_test.out.txt
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.findBootstrapClass(Native Method)
at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:927)
at java.lang.ClassLoader.loadClass(ClassLoader.java:298)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.tools.ant.DefaultLogger.formatTime(DefaultLogger.java:323)
at org.apache.tools.ant.DefaultLogger.buildFinished(DefaultLogger.java:170)
at org.apache.tools.ant.Project.fireBuildFinished(Project.java:2037)
at org.apache.tools.ant.Main.runBuild(Main.java:778)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
PermGen space

Hoss Man
added a comment - 29/Mar/12 03:33 Attaching full stderr/stdout of a (differnet "ant -verbose clean test" run that ended with another PermGen OOM...
hossman@bester:~/lucene/branch_lucene3930$ ant -verbose clean test 2>&1 > ant_-verbose_clean_test.out.txt
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.findBootstrapClass(Native Method)
at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:927)
at java.lang.ClassLoader.loadClass(ClassLoader.java:298)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.tools.ant.DefaultLogger.formatTime(DefaultLogger.java:323)
at org.apache.tools.ant.DefaultLogger.buildFinished(DefaultLogger.java:170)
at org.apache.tools.ant.Project.fireBuildFinished(Project.java:2037)
at org.apache.tools.ant.Main.runBuild(Main.java:778)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
PermGen space

Now we have the problematic patched jars and the renamed unreleased jars.

For the unreleased jars, can we just include their source in our source? This would work for commons-csv and noggit which are the only two in solr/lib. They're both very small jars with only a handful of java files.

I'm running into bigger problems with the libs in the solr contrib langid. The langdect jar is from a revision I cannot find in its googlecode project (due to a switch from svn to git) and the jsonic jar it depends on isn't available in ivy accessible repos, only version 1.2.8 is.

Chris Male
added a comment - 29/Mar/12 03:45 Now we have the problematic patched jars and the renamed unreleased jars.
For the unreleased jars, can we just include their source in our source? This would work for commons-csv and noggit which are the only two in solr/lib. They're both very small jars with only a handful of java files.
I'm running into bigger problems with the libs in the solr contrib langid. The langdect jar is from a revision I cannot find in its googlecode project (due to a switch from svn to git) and the jsonic jar it depends on isn't available in ivy accessible repos, only version 1.2.8 is.

as far as the PermGen OOM goes, the -verbose logs show repeated instances of "Trying to override old definition of task antlib:org.apache.ivy.ant:..." suggesting that rmuir's unless="ivy.uptodate" isn't working the way we think it should (possibly because of the way the various build files are included in one another?)

If you are defining tasks or types that share the same classpath with multiple taskdef or typedef tasks, the corresponding classes will be loaded by different Java ClassLoaders. Two classes with the same name loaded via different ClassLoaders are not the same class from the point of view of the Java VM, they don't share static variables and instances of these classes can't access private methods or attributes of instances defined by "the other class" of the same name. They don't even belong to the same Java package and can't access package private code, either.

The best way to load several tasks/types that are supposed to cooperate with each other via shared Java code is to use the resource attribute and an antlib descriptor. If this is not possible, the second best option is to use the loaderref attribute and specify the same name for each and every typedef/taskdef - this way the classes will share the same ClassLoader. Note that the typedef/taskdef tasks must use identical classpath defintions (this includes the order of path components) for the loaderref attribute to work.

Hoss Man
added a comment - 29/Mar/12 04:02 as far as the PermGen OOM goes, the -verbose logs show repeated instances of "Trying to override old definition of task antlib:org.apache.ivy.ant:..." suggesting that rmuir's unless="ivy.uptodate" isn't working the way we think it should (possibly because of the way the various build files are included in one another?)
If we can't keep the <taskdef/ from running multiple times, we might be able to force it to use the same class loader each time, per the ant docs...
https://ant.apache.org/manual/Tasks/typedef.html
If you are defining tasks or types that share the same classpath with multiple taskdef or typedef tasks, the corresponding classes will be loaded by different Java ClassLoaders. Two classes with the same name loaded via different ClassLoaders are not the same class from the point of view of the Java VM, they don't share static variables and instances of these classes can't access private methods or attributes of instances defined by "the other class" of the same name. They don't even belong to the same Java package and can't access package private code, either.
The best way to load several tasks/types that are supposed to cooperate with each other via shared Java code is to use the resource attribute and an antlib descriptor. If this is not possible, the second best option is to use the loaderref attribute and specify the same name for each and every typedef/taskdef - this way the classes will share the same ClassLoader. Note that the typedef/taskdef tasks must use identical classpath defintions (this includes the order of path components) for the loaderref attribute to work.
...it appears it's just some unique string key to name the classloader? (no idea if it whatever is causing hte current problem will still plague by creating multiple loaders with the same name)...
https://svn.apache.org/viewvc/ant/core/tags/ANT_171/src/etc/testcases/core/loaderref/

Chris Male
added a comment - 29/Mar/12 05:49 Proof of concept patch to show how we can download jetty, patch it, build it and copy the jars of interest into lib.
I've made this against the 3930 branch but its targeting 3x of course.

as far as the PermGen OOM goes, the -verbose logs show repeated instances of "Trying to override old definition of task antlib:org.apache.ivy.ant:..." suggesting that rmuir's unless="ivy.uptodate" isn't working the way we think it should (possibly because of the way the various build files are included in one another?)

Robert Muir
added a comment - 29/Mar/12 07:56
as far as the PermGen OOM goes, the -verbose logs show repeated instances of "Trying to override old definition of task antlib:org.apache.ivy.ant:..." suggesting that rmuir's unless="ivy.uptodate" isn't working the way we think it should (possibly because of the way the various build files are included in one another?)
Thats right... I don't have a good solution here yet.

Robert Muir
added a comment - 29/Mar/12 08:35 I read up enough on that issue to know I don't want to be screwing around with it.
So you have to install ivy in your ~/.ant/lib: I removed any automagical shit, and this is now solved.

I personally don't like it when I need to install ant dependencies in a global scope – this may not be a problem from one project's perspective but if you're working on multiple projects then this can result in global dependencies shadowing project's local definitions and debugging this is a pain.

Not to mention it's another requirement after checkout. I won't be able to look into this now, just expressing my opinion.

Dawid Weiss
added a comment - 29/Mar/12 08:54 So you have to install ivy in your ~/.ant/lib
I personally don't like it when I need to install ant dependencies in a global scope – this may not be a problem from one project's perspective but if you're working on multiple projects then this can result in global dependencies shadowing project's local definitions and debugging this is a pain.
Not to mention it's another requirement after checkout. I won't be able to look into this now, just expressing my opinion.

Proof of concept patch to show how we can download jetty, patch it, build it and copy the jars of interest into lib.

Yeah, thats how we should do it. Long term, for developers (like committers) that do this every day,
we can add a property you can define in your ~/build.properties declaring where a patched jar already exists
so it doesn't slow us down: but thats an optimization for guys like us.

I'm running into bigger problems with the libs in the solr contrib langid. The langdect jar is from a revision I cannot find in its googlecode project (due to a switch from svn to git) and the jsonic jar it depends on isn't available in ivy accessible repos, only version 1.2.8 is.

Ouch, i didn't notice they changed over to git... The git repository they have just doesn't correspond
to any meaningful rev numbers (die git die), but still has the history from svn. We are going to
have to do a git checkout i think?

Robert Muir
added a comment - 29/Mar/12 08:54
Proof of concept patch to show how we can download jetty, patch it, build it and copy the jars of interest into lib.
Yeah, thats how we should do it. Long term, for developers (like committers) that do this every day,
we can add a property you can define in your ~/build.properties declaring where a patched jar already exists
so it doesn't slow us down: but thats an optimization for guys like us.
I'm running into bigger problems with the libs in the solr contrib langid. The langdect jar is from a revision I cannot find in its googlecode project (due to a switch from svn to git) and the jsonic jar it depends on isn't available in ivy accessible repos, only version 1.2.8 is.
Ouch, i didn't notice they changed over to git... The git repository they have just doesn't correspond
to any meaningful rev numbers (die git die), but still has the history from svn. We are going to
have to do a git checkout i think?

I personally don't like it when I need to install ant dependencies in a global scope – this may not be a problem from one project's perspective but if you're working on multiple projects then this can result in global dependencies shadowing project's local definitions and debugging this is a pain.

Not to mention it's another requirement after checkout. I won't be able to look into this now, just expressing my opinion.

There's nothing I can do about this. I spent all day trying to look for a better solution.
You can use ant -lib too if you want, nothing stops you from doing that.

But we have to remove all these jars, and for the build to still work. I'm sure ill piss a lot of people off.

Robert Muir
added a comment - 29/Mar/12 09:02
I personally don't like it when I need to install ant dependencies in a global scope – this may not be a problem from one project's perspective but if you're working on multiple projects then this can result in global dependencies shadowing project's local definitions and debugging this is a pain.
Not to mention it's another requirement after checkout. I won't be able to look into this now, just expressing my opinion.
There's nothing I can do about this. I spent all day trying to look for a better solution.
You can use ant -lib too if you want, nothing stops you from doing that.
But we have to remove all these jars, and for the build to still work. I'm sure ill piss a lot of people off.

Robert Muir
added a comment - 29/Mar/12 09:13
Again – can we make it project-wide ($
Unknown macro: {project.home}
/build.properties) or at least have a unique name (~/lucene-solr.properties)?
its always been:
<!-- Give user a chance to override without editing this file
(and without typing -D each time it compiles it -->
<property file="${user.home}/lucene.build.properties"/>
<property file="${user.home}/build.properties"/>
<property file="${basedir}/build.properties"/>
<property file="${common.dir}/build.properties"/>
So just use lucene.build.properties if you want that.

You can use ant -lib too if you want, nothing stops you from doing that.

Sure, but this introduces additional complexity in setting up shell aliases or simply remembering about it. This just feels wrong somehow (making life harder instead of simpler). I may take a look at it at some point but I also remember having difficulties with multiple antlib definitions in our stuff and I believe you it's a pain.

Dawid Weiss
added a comment - 29/Mar/12 09:14 You can use ant -lib too if you want, nothing stops you from doing that.
Sure, but this introduces additional complexity in setting up shell aliases or simply remembering about it. This just feels wrong somehow (making life harder instead of simpler). I may take a look at it at some point but I also remember having difficulties with multiple antlib definitions in our stuff and I believe you it's a pain.

Sure, but this introduces additional complexity in setting up shell aliases or simply remembering about it. This just feels wrong somehow (making life harder instead of simpler). I may take a look at it at some point but I also remember having difficulties with multiple antlib definitions in our stuff and I believe you it's a pain.

Right but honestly, while it might not be ideal, I don't think its a blocker? Having jars in our source release is. Having the build OOM is. So I made a tradeoff.

If you (or any one else) has the time to look at stuff like this, it would be really appreciated: thats why i created a branch:

We likely also won't have any of the IDE configurations working unless someone has the time to jump into that, and I have no idea if the maven build will break by us not having jars in the source tree. These are all things I will ignore without hesitation because they don't need to block a release. If anyone wants to fix this stuff, just start committing to the branch, i need your help... its just that simple

Robert Muir
added a comment - 29/Mar/12 09:21
Sure, but this introduces additional complexity in setting up shell aliases or simply remembering about it. This just feels wrong somehow (making life harder instead of simpler). I may take a look at it at some point but I also remember having difficulties with multiple antlib definitions in our stuff and I believe you it's a pain.
Right but honestly, while it might not be ideal, I don't think its a blocker? Having jars in our source release is. Having the build OOM is. So I made a tradeoff.
If you (or any one else) has the time to look at stuff like this, it would be really appreciated: thats why i created a branch:
https://svn.apache.org/repos/asf/lucene/dev/branches/lucene3930
We likely also won't have any of the IDE configurations working unless someone has the time to jump into that, and I have no idea if the maven build will break by us not having jars in the source tree. These are all things I will ignore without hesitation because they don't need to block a release. If anyone wants to fix this stuff, just start committing to the branch, i need your help... its just that simple

Dawid Weiss
added a comment - 29/Mar/12 09:36 Having jars in our source release is.
Is a requirement that the source release must "build out of the box"? Or is it about code publication only? I don't know, I'm asking. This seems like a revolutionary build change before the release
Having the build OOM is. So I made a tradeoff.
I understand this, but my gut feeling still says that if you need to install ivy in an ant global space you might as well set ANT_OPTS to increase permgen... The tradeoff made is one of many.
i need your help... its just that simple
Can't jump into it right now, sorry. I'll take a look when I get a spare cycle though. I'm not sure it can be fixed but I'll take a look.

Is a requirement that the source release must "build out of the box"? Or is it about code publication only? I don't know, I'm asking. This seems like a revolutionary build change before the release

Yeah you are telling me. trust me i'm not happy. see my note to the dev-list:

we have an existing stable build/packaging tasks that works, and have been iterated on for 5 3.x release already.

personally i've already tested the crap out of this branch before RC, its not just jenkins but i have my own (http://sierranevada.servebeer.com/) still running now. I was planning on cutting an RC yesterday before this shit
came up.

So i didnt just rush into this shit and make it blocker as some knee-jerk reaction. I don't feel i have any
other choice.

As i said on my email, given the discussion about this on the other thread, i'd rather fix shit than argue
on mailing lists about whether our release is legal or not. If anyone else has a different opinion and is
willing to sacrifice days of their life arguing with apache board members instead, then stand up and we'll
go with whats in branch_3x now.

But given the discussion, personally I'm not going to be involved in (and will vote against) any release
that has jar files in the source.tgz.

Robert Muir
added a comment - 29/Mar/12 09:42
Is a requirement that the source release must "build out of the box"? Or is it about code publication only? I don't know, I'm asking. This seems like a revolutionary build change before the release
Yeah you are telling me. trust me i'm not happy. see my note to the dev-list:
we have an existing stable build/packaging tasks that works, and have been iterated on for 5 3.x release already.
personally i've already tested the crap out of this branch before RC, its not just jenkins but i have my own ( http://sierranevada.servebeer.com/ ) still running now. I was planning on cutting an RC yesterday before this shit
came up.
So i didnt just rush into this shit and make it blocker as some knee-jerk reaction. I don't feel i have any
other choice.
As i said on my email, given the discussion about this on the other thread, i'd rather fix shit than argue
on mailing lists about whether our release is legal or not. If anyone else has a different opinion and is
willing to sacrifice days of their life arguing with apache board members instead, then stand up and we'll
go with whats in branch_3x now.
But given the discussion, personally I'm not going to be involved in (and will vote against) any release
that has jar files in the source.tgz.

Robert Muir
added a comment - 29/Mar/12 10:10
FWIW.... i did a completley clean checkout of the lucene3930 (r1306662) and got the following build failure trying to run "ant clean test" from the top level.
Hoss, thank you for checking out the branch and doing tests! this is really useful, i know
there are more nasties hanging out there...

For the unreleased jars, can we just include their source in our source? This would work for commons-csv and noggit which are the only two in solr/lib. They're both very small jars with only a handful of java files.

Any opinions on this? Do I need to change the packages for anything that we put into our source?

Ouch, i didn't notice they changed over to git... The git repository they have just doesn't correspond
to any meaningful rev numbers (die git die), but still has the history from svn. We are going to
have to do a git checkout i think?

A checkout of head and upgrade to use that? or do a checkout of an older revision? We're going to have to checkout the jsonic source as well.

We likely also won't have any of the IDE configurations working unless someone has the time to jump into that, and I have no idea if the maven build will break by us not having jars in the source tree. These are all things I will ignore without hesitation because they don't need to block a release. If anyone wants to fix this stuff, just start committing to the branch, i need your help... its just that simple

I think both IDE support and Maven will be fine. The idea and eclipse targets will need to invoke resolve to ensure the libs are downloaded. Once they're in the same place as they were before, the IDE configurations should march along fine. Any libs we've changed the name of or dropped, we can just update the configs for. For Maven all this work is actually going to simplify the poms. We just need to bring the dependencies inline with the libs we have. Both wont be big deals.

Chris Male
added a comment - 29/Mar/12 11:31
For the unreleased jars, can we just include their source in our source? This would work for commons-csv and noggit which are the only two in solr/lib. They're both very small jars with only a handful of java files.
Any opinions on this? Do I need to change the packages for anything that we put into our source?
Ouch, i didn't notice they changed over to git... The git repository they have just doesn't correspond
to any meaningful rev numbers (die git die), but still has the history from svn. We are going to
have to do a git checkout i think?
A checkout of head and upgrade to use that? or do a checkout of an older revision? We're going to have to checkout the jsonic source as well.
We likely also won't have any of the IDE configurations working unless someone has the time to jump into that, and I have no idea if the maven build will break by us not having jars in the source tree. These are all things I will ignore without hesitation because they don't need to block a release. If anyone wants to fix this stuff, just start committing to the branch, i need your help... its just that simple
I think both IDE support and Maven will be fine. The idea and eclipse targets will need to invoke resolve to ensure the libs are downloaded. Once they're in the same place as they were before, the IDE configurations should march along fine. Any libs we've changed the name of or dropped, we can just update the configs for. For Maven all this work is actually going to simplify the poms. We just need to bring the dependencies inline with the libs we have. Both wont be big deals.

Robert Muir
added a comment - 29/Mar/12 11:36
A checkout of head and upgrade to use that? or do a checkout of an older revision? We're going to have to checkout the jsonic source as well.
I don't care as long as it works, and as long as its not 'head' but a specific revision/hashcode/whatever so its not changing...

Robert Muir
added a comment - 29/Mar/12 11:39 And for reference, the reason we used the revision we had for that, is so that we
would load the language profiles from the classpath and not the filesystem.
In other words, we need this change: http://code.google.com/p/language-detection/issues/detail?id=24

For git the revision md5 is unique and you can always do a checkout of a particular revision (typically using so-called detached head). This just moves you to a particular version in the revision tree.

Dawid Weiss
added a comment - 29/Mar/12 12:52 For git the revision md5 is unique and you can always do a checkout of a particular revision (typically using so-called detached head). This just moves you to a particular version in the revision tree.

Chris Male
added a comment - 29/Mar/12 12:54 Patch which places noggit and commons csv under org.apache.solr.internal.* and updates everything.
Will definitely need to include a package.html for the internal package detailing its role.

As i said on my email, given the discussion about this on the other thread, i'd rather fix shit than argue
on mailing lists about whether our release is legal or not. If anyone else has a different opinion and is
willing to sacrifice days of their life arguing with apache board members instead, then stand up and we'll
go with whats in branch_3x now.

Robert Muir
added a comment - 29/Mar/12 13:32
As i said on my email, given the discussion about this on the other thread, i'd rather fix shit than argue
on mailing lists about whether our release is legal or not. If anyone else has a different opinion and is
willing to sacrifice days of their life arguing with apache board members instead, then stand up and we'll
go with whats in branch_3x now.

Yonik Seeley
added a comment - 29/Mar/12 15:06 if anyone else has a different opinion and is willing to sacrifice days of their life arguing with apache board members instead
The third option was to do nothing hasty and wait for the actual board to come to a decision and communicate with PMCs. Too late for this issue... but something to consider for the future.

Robert Muir
added a comment - 29/Mar/12 15:25 Too late for this issue, because we are doing all the work, and you are just whining.
Meanwhile, the noggit problem you are bitching about is in fact 100% your fault. Get over your control issues and make it a real open source project with real releases already.

Moving C2 JARs out means we will have to re-release an archival version of C2 just for this purpose. The reasons are that we depend on libraries which themselves don't have 1.5 equivalents (mahout-math) so any newer version would have to be re-released along with backcompat of these libraries too... long story.

Anyway, I will release a weird-looking version 3.5.0.1 which will be 1.5 compatible. I will let you know (and possibly modify the branch) once this happens. Give me a few hours, it'll require some checks/ testing.

Dawid Weiss
added a comment - 29/Mar/12 15:41 Moving C2 JARs out means we will have to re-release an archival version of C2 just for this purpose. The reasons are that we depend on libraries which themselves don't have 1.5 equivalents (mahout-math) so any newer version would have to be re-released along with backcompat of these libraries too... long story.
Anyway, I will release a weird-looking version 3.5.0.1 which will be 1.5 compatible. I will let you know (and possibly modify the branch) once this happens. Give me a few hours, it'll require some checks/ testing.

If the mountain won't come to us, we should at least make it easier for users to get to the mountain.

attached patch fails with a helpful message if the ivy tasks can't be loaded from the ant classpath, and adds a new bootstrap target users can run to have us install ivy into ~/.ant/lib for them if they want

I'm sure a lot of improvements could be made to the wording of the helpful message .. it's probably got some typos in it.

hossman@bester:~/lucene/branch_lucene3930$ ant compile
Buildfile: build.xml
compile:
compile-lucene-core:
jflex-uptodate-check:
jflex-notice:
javacc-uptodate-check:
javacc-notice:
ivy-availablity-check:
[echo]
[echo] This build requires Ivy and Ivy could not be found in your ant classpath
[echo]
[echo] (Due to classpath issues and the recursive nature of the Lucene/Solr
[echo] build system, a local copy of Ivy can not be used an loaded dynamicaly
[echo] by the build.xml)
[echo]
[echo] You can either manually install a copy of Ivy 2.2.0 in your ant classpath:
[echo] http://ant.apache.org/manual/install.html#optionalTasks
[echo]
[echo] Or this build file can do it for you by running the Ivy Bootstrap target:
[echo] ant ivy-bootstrap
[echo]
[echo] Either way you will only have to install Ivy one time.
[echo]
[echo] 'ant ivy-bootstrap' will install a copy of Ivy into your Ant User Library:
[echo] /home/hossman/.ant/lib
[echo]
[echo] If you would prefer, you can have it installed into an alternative
[echo] directory using the "-Divy_install_path=/some/path/you/choose" option,
[echo] but you will have to specify this path every time you build Lucene/Solr
[echo] in the future...
[echo] ant ivy-bootstrap -Divy_install_path=/some/path/you/choose
[echo] ...
[echo] ant -lib /some/path/you/choose clean compile
[echo] ...
[echo] ant -lib /some/path/you/choose clean compile
[echo]
ivy-fail:
BUILD FAILED
/home/hossman/lucene/branch_lucene3930/build.xml:52: The following error occurred while executing this line:
/home/hossman/lucene/branch_lucene3930/lucene/common-build.xml:561: The following error occurred while executing this line:
/home/hossman/lucene/branch_lucene3930/lucene/common-build.xml:289: Ivy is not available
Total time: 1 second

Hoss Man
added a comment - 29/Mar/12 20:55 If the mountain won't come to us, we should at least make it easier for users to get to the mountain.
attached patch fails with a helpful message if the ivy tasks can't be loaded from the ant classpath, and adds a new bootstrap target users can run to have us install ivy into ~/.ant/lib for them if they want
I'm sure a lot of improvements could be made to the wording of the helpful message .. it's probably got some typos in it.
hossman@bester:~/lucene/branch_lucene3930$ ant compile
Buildfile: build.xml
compile:
compile-lucene-core:
jflex-uptodate-check:
jflex-notice:
javacc-uptodate-check:
javacc-notice:
ivy-availablity-check:
[echo]
[echo] This build requires Ivy and Ivy could not be found in your ant classpath
[echo]
[echo] (Due to classpath issues and the recursive nature of the Lucene/Solr
[echo] build system, a local copy of Ivy can not be used an loaded dynamicaly
[echo] by the build.xml)
[echo]
[echo] You can either manually install a copy of Ivy 2.2.0 in your ant classpath:
[echo] http://ant.apache.org/manual/install.html#optionalTasks
[echo]
[echo] Or this build file can do it for you by running the Ivy Bootstrap target:
[echo] ant ivy-bootstrap
[echo]
[echo] Either way you will only have to install Ivy one time.
[echo]
[echo] 'ant ivy-bootstrap' will install a copy of Ivy into your Ant User Library:
[echo] /home/hossman/.ant/lib
[echo]
[echo] If you would prefer, you can have it installed into an alternative
[echo] directory using the "-Divy_install_path=/some/path/you/choose" option,
[echo] but you will have to specify this path every time you build Lucene/Solr
[echo] in the future...
[echo] ant ivy-bootstrap -Divy_install_path=/some/path/you/choose
[echo] ...
[echo] ant -lib /some/path/you/choose clean compile
[echo] ...
[echo] ant -lib /some/path/you/choose clean compile
[echo]
ivy-fail:
BUILD FAILED
/home/hossman/lucene/branch_lucene3930/build.xml:52: The following error occurred while executing this line:
/home/hossman/lucene/branch_lucene3930/lucene/common-build.xml:561: The following error occurred while executing this line:
/home/hossman/lucene/branch_lucene3930/lucene/common-build.xml:289: Ivy is not available
Total time: 1 second

However, re-running the same command then got further. But then it froze during jetty-javadoc download and I had to kill and restart. Don't know what happened, have not tried to reproduce. On third attempt all dependencies were fetched and the build succeeded.

Questions:

For solr/contrib/extraction, could not tika-parsers use transitive="true" to avoid specifying versions for all parsers?

Why do we let ivy download sources and javadoc jars? They can be excluded in ivy.xml

Jan Høydahl
added a comment - 29/Mar/12 23:40 Guys, this is impressive piece of heavy committing!
I tested the build on my MacBook. I love how the checkout is small and how it downloads everything on demand!
First time I ran ant example the build failed with
[ivy:retrieve] [FAILED ] com.ibm.icu#icu4j;4.8.1.1!icu4j.jar: Downloaded file size doesn't match expected Content Length for http://repo1.maven.org/maven2/com/ibm/icu/icu4j/4.8.1.1/icu4j-4.8.1.1.jar. Please retry. (204243ms)
However, re-running the same command then got further. But then it froze during jetty-javadoc download and I had to kill and restart. Don't know what happened, have not tried to reproduce. On third attempt all dependencies were fetched and the build succeeded.
Questions:
For solr/contrib/extraction, could not tika-parsers use transitive="true" to avoid specifying versions for all parsers?
Why do we let ivy download sources and javadoc jars? They can be excluded in ivy.xml

Robert Muir
added a comment - 29/Mar/12 23:44
For solr/contrib/extraction, could not tika-parsers use transitive="true" to avoid specifying versions for all parsers?
Because by turning it off always, we know exactly what jars we have from the ivy.xml. I think this is the way to go.
Why do we let ivy download sources and javadoc jars? They can be excluded in ivy.xml
Just optimizations that havent been done yet... no reason really.

For solr/contrib/extraction, could not tika-parsers use transitive="true" to avoid specifying versions for all parsers?

Having been the one to go through and add all those parsers and versions, there were many times I wished I did use transitive. But Robert is right, at this stage we get to see clearly which jars we're using. In the future I can perhaps see places in Solr for using transitive, but in Lucene I think we should just be clear what dependencies we have.

Why do we let ivy download sources and javadoc jars? They can be excluded in ivy.xml

I quite like this because then we have the sources and javadocs when developing. They're downloaded once per dependency, no biggie?

Chris Male
added a comment - 30/Mar/12 00:05
For solr/contrib/extraction, could not tika-parsers use transitive="true" to avoid specifying versions for all parsers?
Having been the one to go through and add all those parsers and versions, there were many times I wished I did use transitive. But Robert is right, at this stage we get to see clearly which jars we're using. In the future I can perhaps see places in Solr for using transitive, but in Lucene I think we should just be clear what dependencies we have.
Why do we let ivy download sources and javadoc jars? They can be excluded in ivy.xml
I quite like this because then we have the sources and javadocs when developing. They're downloaded once per dependency, no biggie?

Jan Høydahl
added a comment - 30/Mar/12 01:15 The attached patch disables fetching of source jars and javadoc jars by default, to optimize for speed and bandwidth. Without it, the build takes much longer time.
But it is nice to have the ability to get them automatically. So the exclude is parameterized You can enable fetching by specifying a Java option (only needed once)
ant -Dfetch.sources.javadocs compile

The only real remaining issue here is how to handle the langdetect jar in Solr's langid contrib. The solution in ivy works fine (specifying the location of the jar) except I don't think you can do the same thing in Maven, which stumps our Maven releases a little. For some people that isn't an issue, but I'd like to see it solved. Does anybody know of how to do something similar in Maven?

The other alternative is to get the language-detection project to make an actual release. It is possible to simulate a repo in a googlecode project. But I'm unsure how to go about getting the project maintainer to do that?

Chris Male
added a comment - 30/Mar/12 10:17 The only real remaining issue here is how to handle the langdetect jar in Solr's langid contrib. The solution in ivy works fine (specifying the location of the jar) except I don't think you can do the same thing in Maven, which stumps our Maven releases a little. For some people that isn't an issue, but I'd like to see it solved. Does anybody know of how to do something similar in Maven?
The other alternative is to get the language-detection project to make an actual release. It is possible to simulate a repo in a googlecode project. But I'm unsure how to go about getting the project maintainer to do that?

Don't we need to get rid of the binary JAR anyway? If so, the alternatives are to either put all the sources in lucene repo or push a maven release of that JAR. SonaType accepts third-party JAR pushes too – one can do it as a last resort option.

Dawid Weiss
added a comment - 30/Mar/12 10:27 Don't we need to get rid of the binary JAR anyway? If so, the alternatives are to either put all the sources in lucene repo or push a maven release of that JAR. SonaType accepts third-party JAR pushes too – one can do it as a last resort option.
https://docs.sonatype.org/display/Repository/Uploading+3rd-party+Artifacts+to+The+Central+Repository

For our official ant build, the jar is now retrieved as an ivy dependency so we've gotten rid of the binary jar. But its done so using what I think is an ivy only mechanism of specifying a path to a jar. The language-detection project fortunately includes a jar of itself in its own git, so we simply pull that jar. But we can't do that for Maven, I think? So yes, we need a release.

I hadn't seen the Sonatype thing before, that is certainly worth considering.

Chris Male
added a comment - 30/Mar/12 10:39 For our official ant build, the jar is now retrieved as an ivy dependency so we've gotten rid of the binary jar. But its done so using what I think is an ivy only mechanism of specifying a path to a jar. The language-detection project fortunately includes a jar of itself in its own git, so we simply pull that jar. But we can't do that for Maven, I think? So yes, we need a release.
I hadn't seen the Sonatype thing before, that is certainly worth considering.

You can do it with Maven by specifying an optional system dependency off the project's basedir and fetching the JAR in a preliminary phase... I think. But it's a hack beyond dirty. And it doesn't make other people's lives any easier (if somebody uses your pom).

Dawid Weiss
added a comment - 30/Mar/12 10:54 You can do it with Maven by specifying an optional system dependency off the project's basedir and fetching the JAR in a preliminary phase... I think. But it's a hack beyond dirty. And it doesn't make other people's lives any easier (if somebody uses your pom).

Chris Male
added a comment - 30/Mar/12 11:14 I'm adding a message on the project to see if the maintainer's prepared to following Sonatype's official maven release process. If not, then we can go the 3rd party route.

Robert Muir
added a comment - 30/Mar/12 11:18 keep in mind the solution i have only works for trunk, because they compile with java 6.
their code compiles just fine with a java5 compiler or with java 5 options...
from ivy perspective i can add ant logic to just check out their source and compile it ourselves.
but just keep it in mind for maven.

If we go the third party route I suggest to release an artifact with a -jdk15 classifier to make it explicit it's a 1.5 build. Perhaps we can suggest to the maintainer to compile with 1.5 compatibility if this doesn't involve any source code changes?

Dawid Weiss
added a comment - 30/Mar/12 11:31 If we go the third party route I suggest to release an artifact with a -jdk15 classifier to make it explicit it's a 1.5 build. Perhaps we can suggest to the maintainer to compile with 1.5 compatibility if this doesn't involve any source code changes?

Robert Muir
added a comment - 30/Mar/12 11:39
https://docs.sonatype.org/display/Repository/Uploading+3rd-party+Artifacts+to+The+Central+Repository
Thanks for bringing this up dawid. Why didn't anyone bring this up before?
Is it considered a dark, secret option that disrupts the viral natural of maven?
I can't believe we've had maven artifacts like solr-xxx for all these 3.x releases
when there was an official mechanism to do this: 'as long as the license permits'
as the page says.

I hadn't ever heard about it to be honest. Examining discussions on the language-detection project, the maintainer expressed an inability to find time to make a Maven release, but that was about 9 months ago. I've dropped in a comment seeing if he'd be prepared to do the release, if not, we know what we have to do.

Chris Male
added a comment - 30/Mar/12 11:51 I hadn't ever heard about it to be honest. Examining discussions on the language-detection project, the maintainer expressed an inability to find time to make a Maven release, but that was about 9 months ago. I've dropped in a comment seeing if he'd be prepared to do the release, if not, we know what we have to do.

Robert Muir
added a comment - 30/Mar/12 12:22
The attached patch disables fetching of source jars and javadoc jars by default, to optimize for speed and bandwidth. Without it, the build takes much longer time.
Jan do you want to commit this? I think its a nice improvement! I can commit it for you if you don't have the time!

I added top-level/recursive resolve tasks to resolve any deps and fixed ant eclipse to depend on this.
Also added 'clean-jars' top-level that basically does rm */.jar
I would add this before the hudson 'svn status' check, so if someone commits any jars to the source tree it fails.

At this point I think everything is generally working nice in trunk (compile/test/packaging/javadocs/maven/...)

I'll be doing testing of all the tasks from all the modules today to find any kinks, but I think we are nearly ready
to land in trunk.

I'll look at intellij (i already made it depend on resolve) and see if i can get it into shape, since
e.g. junit.jar and friends are now under test-framework/lib.

Robert Muir
added a comment - 30/Mar/12 14:44 I added top-level/recursive resolve tasks to resolve any deps and fixed ant eclipse to depend on this.
Also added 'clean-jars' top-level that basically does rm * / .jar
I would add this before the hudson 'svn status' check, so if someone commits any jars to the source tree it fails.
At this point I think everything is generally working nice in trunk (compile/test/packaging/javadocs/maven/...)
I'll be doing testing of all the tasks from all the modules today to find any kinks, but I think we are nearly ready
to land in trunk.
I'll look at intellij (i already made it depend on resolve) and see if i can get it into shape, since
e.g. junit.jar and friends are now under test-framework/lib.
Any help testing or fixing intellij or whatever is appreciated

Robert Muir
added a comment - 30/Mar/12 15:15 ok i downloaded intellij, and from what i can tell its working now...
(I can run lucene and solr tests etc and it doesnt give me any errors that indicate any issues)
i'll be testing the various ant tasks and trying to find any bugs now.

Robert Muir
added a comment - 30/Mar/12 16:31 i spent about an hour blowing away everything (ant clean clean-jars) and running
various ant targets from various locations and checking stuff out.
In my opinion this is ready to go into trunk. I'll wait a bit for any feedback though.

Ryan McKinley
added a comment - 30/Mar/12 17:16 In my opinion this is ready to go into trunk. I'll wait a bit for any feedback though.
+1 ant test worked on my first try (after ivy bootstrap)
HUGE thanks for looking into this

I think later, we should investigate pulling some of the large test data stuff,
like the 5MB europarl-lines file via ivy, this is easy enough to do, would be
cached locally etc, and make the source release much smaller.

This could also make it possible to use the 'large' one easier that jenkins uses...

Robert Muir
added a comment - 30/Mar/12 17:29 Just some stats on source release sizes:
Trunk:
-rw-rw-r-- 1 rmuir rmuir 11372783 2012-03-30 12:25 lucene-4.0-SNAPSHOT-src.tgz
-rw-rw-r-- 1 rmuir rmuir 107074928 2012-03-30 12:20 apache-solr-4.0-SNAPSHOT-src.tgz
Branch:
-rw-rw-r-- 1 rmuir rmuir 8207455 2012-03-30 12:26 lucene-4.0-SNAPSHOT-src.tgz
-rw-rw-r-- 1 rmuir rmuir 30676680 2012-03-30 12:23 apache-solr-4.0-SNAPSHOT-src.tgz
I think later, we should investigate pulling some of the large test data stuff,
like the 5MB europarl-lines file via ivy, this is easy enough to do, would be
cached locally etc, and make the source release much smaller.
This could also make it possible to use the 'large' one easier that jenkins uses...
But those are optimizations for later issues...

I'll work on merging this back. I've tried to setup jenkins to where there will be no failed builds,
but its the kind of thing where i'll be surprised if I get away without a failure. If there are problems,
I will fix. If the problems are serious, I will revert.

As far as the plan for 3.x: my idea is we bake in trunk for a while (let jenkins have its way with the
change, etc). At the same time we can look at how to backport for hopefully next week.

I might create a branch off of 3.x (the same way as this one worked) so that its easier for people to help.
But first lets get it in trunk and see how that goes.

Robert Muir
added a comment - 30/Mar/12 18:27 I'll work on merging this back. I've tried to setup jenkins to where there will be no failed builds,
but its the kind of thing where i'll be surprised if I get away without a failure. If there are problems,
I will fix. If the problems are serious, I will revert.
As far as the plan for 3.x: my idea is we bake in trunk for a while (let jenkins have its way with the
change, etc). At the same time we can look at how to backport for hopefully next week.
I might create a branch off of 3.x (the same way as this one worked) so that its easier for people to help.
But first lets get it in trunk and see how that goes.

One final test, i checked out on laptop (using hossmans bootstrap), and ran 'ant resolve'
(also 'ant idea' or 'ant eclipse' call this).. and then cut internet and ran tests/built
and ran solr example offline. So I think its ready.

Robert Muir
added a comment - 30/Mar/12 18:42 One final test, i checked out on laptop (using hossmans bootstrap), and ran 'ant resolve'
(also 'ant idea' or 'ant eclipse' call this).. and then cut internet and ran tests/built
and ran solr example offline. So I think its ready.

I did some testing of the "packages" built using trunk (circa r1307608)...

we don't ship solr's build.xml (or any of the sub-build.xml files) in the "binary" artifacts, and with these changes most of the new ivy.xml files are also excluded – but for some reason these newly added files are showing up, we should probably figure out why and exclude them as well since they aren't usable and could easily people...

./example/example-DIH/ivy.xml

./example/example-DIH/build.xml

./example/ivy.xml

./example/build.xml

the lib's for test-framework (ant, ant-junit, and junit) aren't being included in the lucene "binary" artifacts ... for the ant jars this might (test-framework doesn't actually have any run-time deps on anything in ant does it?) but it seems like hte junit jar should be included since including lucene-test-framework.jar in your classpath is useless w/o also including junit

"ant ivy-bootstrap" followed by "ant test" using the lucene "source" package (lucene-4.0-SNAPSHOT-src.tgz) produces a build failure – but this may have been a problem even before ivy (note the working dir and the final error)...

categorically exclude build.xml and ivy.xml files from solr binary packages (to prevent the ones under example from being included)

add parity to what files under test-framework get included in line with how contrib is treated (new patterns try to match some things that don't existing in test-framework, but i don't think that's bad – future proofs us)

Hoss Man
added a comment - 31/Mar/12 21:39 patch fixing the first two problems i mentioned above:
categorically exclude build.xml and ivy.xml files from solr binary packages (to prevent the ones under example from being included)
add parity to what files under test-framework get included in line with how contrib is treated (new patterns try to match some things that don't existing in test-framework, but i don't think that's bad – future proofs us)

Jan Høydahl
added a comment - 01/Apr/12 00:20 We have a 7Mb jar which is included in the binary distro twice. Any way to get rid of one?
./contrib/analysis-extras/lib/icu4j-4.8.1.1.jar
./contrib/extraction/lib/icu4j-4.8.1.1.jar
Also, from what I can see, solr/contrib/extraction/lib/xml-apis-1.0.b2.jar dependency is redundant - tests pass without it
See https://issues.apache.org/jira/browse/TIKA-412 and https://issues.apache.org/jira/browse/LUCENE-2961

Robert Muir
added a comment - 01/Apr/12 23:43 Hi guys, thanks for testing!
Can we open separate issues for these remaining things? These problems are unrelated in
any way to any changes in this issue and are pre-existing conditions:
LUCENE-2999 : lucene source release does not work because packaging is fucked
SOLR-2435 : duplicated icu4j jars in binary releases
Its a good thing that packaging is screwed up exactly like it was before this change. Because
i didnt really have to change its logic..., i kept what was there before.
Thanks;

Chris Male
added a comment - 02/Apr/12 05:16 Haven't heard anything back from the language-detection project, so I'm pressing ahead. Attached the pom.xml I've created for the project, for review from anybody who knows this kinda thing.

Looks good to me, Chris. Two minor things:
1) sourceDirectory and testSourceDirectory look like default values anyway?
2) there is a newer version of jsonic in maven repositories; don't know if this matters at all.

Dawid Weiss
added a comment - 02/Apr/12 07:54 Looks good to me, Chris. Two minor things:
1) sourceDirectory and testSourceDirectory look like default values anyway?
2) there is a newer version of jsonic in maven repositories; don't know if this matters at all.

Chris Male
added a comment - 02/Apr/12 08:25 - edited Thanks for the review.
1) sourceDirectory and testSourceDirectory look like default values anyway?
Err.. yes, I used the pom the maintainer had started and forgot to pull those out. Will do.
2) there is a newer version of jsonic in maven repositories; don't know if this matters at all.
I will stick with the version used in the project.

Robert Muir
added a comment - 02/Apr/12 12:41
patch fixing the first two problems i mentioned above:
Thanks Hossman: I committed this (I am looking at the 3.x merge, i don't want to find/fix the bug twice)!

Robert Muir
added a comment - 02/Apr/12 18:07 I merged back to branch_3x. will let hudson chomp on it for a while, then mark this issue resolved
if there are no immediate problems.
Any later improvements can be followup issues.

I can't get either branch_3x or trunk to build now, on a system that used to build branch_3x without complaint. It says that ivy is not available, even after doing "ant ivy-bootstrap" to download ivy into the home directory. Specifically I am trying to build solrj from trunk, but I can't even get "ant" in the root directory of the checkout to work. I'm on CentOS 6 with oracle jdk7 built using the city-fan.org SRPMs. Ant (1.7.1) and junit are installed from package repositories. Building a checkout of lucene_solr_3_5 on the same machine works fine.

Shawn Heisey
added a comment - 03/Apr/12 21:00 I can't get either branch_3x or trunk to build now, on a system that used to build branch_3x without complaint. It says that ivy is not available, even after doing "ant ivy-bootstrap" to download ivy into the home directory. Specifically I am trying to build solrj from trunk, but I can't even get "ant" in the root directory of the checkout to work. I'm on CentOS 6 with oracle jdk7 built using the city-fan.org SRPMs. Ant (1.7.1) and junit are installed from package repositories. Building a checkout of lucene_solr_3_5 on the same machine works fine.

After printing out all the echo statements under ivy-availability, it spits out this:

ivy-fail:

BUILD FAILED
/index/src/trunk/build.xml:42: The following error occurred while executing this line:
/index/src/trunk/lucene/common-build.xml:584: The following error occurred while executing this line:
/index/src/trunk/lucene/common-build.xml:298: Ivy is not available

By adding <echo message="$

{java.class.path}

"/> to the validate section of build.xml, I got it to print out the java classpath, which includes the jar downloaded by the ivy-bootstrap option:

Shawn Heisey
added a comment - 03/Apr/12 21:29 After printing out all the echo statements under ivy-availability, it spits out this:
ivy-fail:
BUILD FAILED
/index/src/trunk/build.xml:42: The following error occurred while executing this line:
/index/src/trunk/lucene/common-build.xml:584: The following error occurred while executing this line:
/index/src/trunk/lucene/common-build.xml:298: Ivy is not available
By adding <echo message="$
{java.class.path}
"/> to the validate section of build.xml, I got it to print out the java classpath, which includes the jar downloaded by the ivy-bootstrap option:
[echo] /usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/share/java/jaxp_parser_impl.jar:/usr/share/java/xml-commons-apis.jar:/usr/share/java/junit.jar:/usr/share/java/ant/ant-junit.jar:/usr/share/java/ant/ant-nodeps.jar:/usr/lib/jvm/java/lib/tools.jar:/home/ncindex/.ant/lib/ivy-2.2.0.jar:/usr/share/ant/lib/ant-bootstrap.jar:/usr/share/ant/lib/ant-junit.jar:/usr/share/ant/lib/ant-nodeps.jar:/usr/share/ant/lib/ant-launcher.jar:/usr/share/ant/lib/ant.jar

Mark Miller
added a comment - 03/Apr/12 21:31 Did you start a new terminal session after doing ant ivy-bootstrap? FWIW, I had to restart eclipse before I could continue using the eclipse/ant integration.

Shawn Heisey
added a comment - 03/Apr/12 22:41 Did you start a new terminal session after doing ant ivy-bootstrap? FWIW, I had to restart eclipse before I could continue using the eclipse/ant integration.
I did not think of that, as I am not using Eclipse or any other IDE. This is purely commandline via ssh. I did open a new ssh session and try again, no change.

Robert Muir
added a comment - 03/Apr/12 22:42 Please take this to the list (This issue is resolved)... but I don't trust the
'ants' shipped on some of these linux os's... best to download your own ant.

Shawn, I have similar problems with the builtin ant on Fedora 13, and when I add that same echo line, I can see that the ~/.ant/lib/ivy-2.2.0.jar is on the CLASSPATH... yet it fails the ivy-availability-check.

I never got to the bottom of it ... but installing ant myself (1.8.2) and using that version instead worked around it...

Michael McCandless
added a comment - 03/Apr/12 22:58 Shawn, I have similar problems with the builtin ant on Fedora 13, and when I add that same echo line, I can see that the ~/.ant/lib/ivy-2.2.0.jar is on the CLASSPATH... yet it fails the ivy-availability-check.
I never got to the bottom of it ... but installing ant myself (1.8.2) and using that version instead worked around it...