Home » Eclipse Projects » Buckminster » Buckminster does not materialize the source for all plugins(Buckminster materializes two versions of org.apache.commons.logging (1.0.4 and 1.1.1), but only the source for one of them and that leads to the subsequent build failure)

You get errors indicating failure to find the org.apache.commons.logging.source bundle using the 'local' readerType in
searchPath 'local'. I can't understand how that would happen given the cquery/rmaps that you attached.

You get errors indicating failure to find the org.apache.commons.logging.source bundle using the 'local' readerType in
searchPath 'local'. I can't understand how that would happen given the cquery/rmaps that you attached.

- thomas

Not sure what errors you refer to.

If you mean the error from "using the TP" to build, then yes, the build uses a local reader and the (previously created) TP only to build and the error is that the org.apache.commons.logging.source bundle is not found by the local reader (as buckminster never mentions the search in the TP). I could present the buckminster config for the feature build, too if necessary, as the configuration files here are only for the hudson job that builds the TP. The other hudson job just uses the (previously built) TP to build a feature (so far unsucessfully).

What I don't understand is why the TP created by the configuration attached lacks some source plugins (org.apache.commons.logging.source is not the only one) for certain plugins.

I am really stuck here, as my build does not work complaining about the missing source plugin and I have no idea how to fix that.

On 2011-08-30 23:28, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Tue, 30 August 2011 13:37
>> You get errors indicating failure to find the org.apache.commons.logging.source bundle using the 'local' readerType in
>> searchPath 'local'. I can't understand how that would happen given the cquery/rmaps that you attached.
>>
>> - thomas
>
>
> Not sure what errors you refer to.
>
> If you mean the error from "using the TP" to build, then yes, the build uses a local reader and the (previously created)
> TP only to build and the error is that the org.apache.commons.logging.source bundle is not found by the local reader (as
> buckminster never mentions the search in the TP). I could present the buckminster config for the feature build, too if
> necessary, as the configuration files here are only for the hudson job that builds the TP. The other hudson job just
> uses the (previously built) TP to build a feature (so far unsucessfully).
>
You referred to another forum thread. I meant the errors in that thread. They don't seem correlated with the buckminster
files that you present here so I have no idea what I'm looking at or what problems that are caused by what files.

> What I don't understand is why the TP created by the configuration attached lacks some source plugins
> (org.apache.commons.logging.source is not the only one) for certain plugins.
>
> I am really stuck here, as my build does not work complaining about the missing source plugin and I have no idea how to
> fix that.
>
Are you using the 'buckminster.download.source=true' when you materialize the target platform?
Have you verified that the repositories actually contain the sources that you are missing?

On 2011-08-30 23:28, Lothar Werzinger wrote:
> I am really stuck here, as my build does not work complaining about the missing source plugin and I have no idea how to
> fix that.
>
Are you using the 'buckminster.download.source=true' when you materialize the target platform?
Have you verified that the repositories actually contain the sources that you are missing?

- thomas

I have attached a zip file containing two hudson jobs.
The first one creates the target platform and the second one tries to build a site.p2 action on a feature.
The jobs are self contained and should allow you to reproduce my problem.

After building the TP you can go to
test_tp/lastSuccessfulBuild/artifact/targetPlatform/plugins/
and you'll see that these org.apache.commons.logging plugins are part of the TP:

What's really puzzling is why the org.apache.commons.logging_1.0.4 get's materialized in the first place, as when I use the "Plug-in Dependencies" view in PDE it's not referenced at all, only org.apache.commons.logging_1.1.1 is:

One thing that you could try is to keep your rmap cleaner. Use only indigo and 3.7 for instance. If you want to base a
product on Helios and 3.6, you could do that too of course but I think trying to do both is asking for trouble.

The failing build I am referring to is using indigo (via the filter mechanism), but I can try to remove the currently unused entries.

Edit:
As expected removing the unused entries in the rmap did not change (improve) the behaviour.

Can you please try to reproduce the problem with the jobs attached to the previous post? I tried many tweaks to the buckminster configuration (including a version override for the logging plugin), but nothing fixes the problem.

Can you please try to reproduce the problem with the jobs attached to the previous post? I tried many tweaks to the buckminster configuration (including a version override for the logging plugin), but nothing fixes the problem.

In your platform.rmap, all your searchPaths are configured with source="false" and mutable="false". Try setting source to true everywhere and mutable to true on the maven provider (thats where the conversion is needed to be done as far as I understand).

Edit: Just tried out your example in IDE (Bucky 3.6), now I'm getting:

Edit 2: The warnings seem to be harmless. Both versions are being materialized but I still only see org.apache.commons.logging.source_1.0.4.v201101211617 in the TP path (resolving is still running though, and I have to go now).

Edit 3: It would be interesting to know where the 1.1.1 version is coming from. To me it looks like the source repo (may it be p2 or maven), does not have proper source associated with it.

In your platform.rmap, all your searchPaths are configured with source="false" and mutable="false". Try setting source to true everywhere and mutable to true on the maven provider (thats where the conversion is needed to be done as far as I understand).

As far as I know source="true" and mutable="true" is only used when one wants to build from source, not for materializing a target platform out of existing bundles.

Philipp Nanz wrote on Thu, 01 September 2011 00:23

Edit 2: The warnings seem to be harmless. Both versions are being materialized but I still only see org.apache.commons.logging.source_1.0.4.v201101211617 in the TP path (resolving is still running though, and I have to go now).

That is exactly my problem. Only 1.0.4 get's a source download, 1.1.1 doesn't.

Philipp Nanz wrote on Thu, 01 September 2011 00:23

Edit 3: It would be interesting to know where the 1.1.1 version is coming from. To me it looks like the source repo (may it be p2 or maven), does not have proper source associated with it.

Like I said in a previous post, the 1.0.4 is the more mysterious one, as it's not referenced by other plugins.

I would not care that it's there if buckminster would also download the source for the 1.1.1, but it doesn't.

On 2011-09-01 19:49, Lothar Werzinger wrote:
> There must be a way to create a valid working target platform out of the official Indigo repositories. If anyone knows
> how to fix this problem, please speak up. Thanks!

We create target platforms for Buckminster, b3, and Geppetto. b3 is perhaps the most complex one and also the only one
that brings in both versions of the org.apache.commons.logging bundle. We don't see this problem though. We get source
bundles for both versions.

On 2011-09-01 18:46, Lothar Werzinger wrote:
> Philipp Nanz wrote on Thu, 01 September 2011 00:23
>> In your platform.rmap, all your searchPaths are configured with source="false" and mutable="false". Try setting source
>> to true everywhere and mutable to true on the maven provider (thats where the conversion is needed to be done as far
>> as I understand).
>
>
> As far as I know source="true" and mutable="true" is only used when one wants to build from source, not for
> materializing a target platform out of existing bundles.
>
Yes, your understanding is correct. You must never use source="true" when populating your TP. Even if the jars in
question contains source, they are still considered binary bundles in this context.

On 2011-09-01 19:49, Lothar Werzinger wrote:
> There must be a way to create a valid working target platform out of the official Indigo repositories. If anyone knows
> how to fix this problem, please speak up. Thanks!

We create target platforms for Buckminster, b3, and Geppetto. b3 is perhaps the most complex one and also the only one
that brings in both versions of the org.apache.commons.logging bundle. We don't see this problem though. We get source
bundles for both versions.

That's exactly what I need, to build a complex TP (with EMF, CDO, BIRT, RAP and even more things in the future).
Most examples like the famous mail example just need RCP and my experience so far is that while the easy configurations are usually working I always seem to run into problems with the complicated ones.

The fact that only one of the org.apache.commons.logging bundles is missing the source is what puzzles me most. As said earlier I could care less if there are two versions of the plugin in the TP as long as it does not create build problems later on.

On 2011-09-06 02:12, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Thu, 01 September 2011 12:52
>> I'll try to find some time to try your setup over the weekend.
>>
>> - thomas
>
>
> Did you have a chance to look into the issue? Any luck finding the cause?
>
Sorry, didn't find any time. Things are really busy right now. I haven't forgotten it though.

That's great news. Will there be a new release? I have my hands full trying the b3 aggregator as you suggested and I would rather not have to build buckminster from source as an extra potential source of trouble.

On 2011-09-12 22:06, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Mon, 12 September 2011 05:19
>> Hi Lothar,
>>
>> I found the cause of this problem and fixed it. You should get rid of the problem if you update to the most recent
>> Buckminster. More info here:
>>
>> Bug 357060 - Generated conflicting source dependencies are sometimes left out
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=357060
>>
>> Regards,
>> Thomas Hallgren
>
>
> That's great news. Will there be a new release?

This fix has already been pushed to our update sites. So as I wrote earlier, you should get rid of the problem by just
updating to the most recent Buckminster.

On 2011-09-13 08:40, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Mon, 12 September 2011 23:10
>> On 2011-09-13 06:30, Lothar Werzinger wrote:
>> > What is the trick to build and create p2 sites that work like the default
>> http://download.eclipse.org/tools/buckminster/headless-3.6 and don't produce errors on install?
>> >
>> Use Buckminster-3.7 at all times.
>>
>> - thomas
>
>
> The hudson buckminster plugin only supports 3.5 and 3.6. That and the latest fix you made were why I decided to build it
> myself, but as outlined I had problems with the created p2 sites.

On 2011-09-13 08:40, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Mon, 12 September 2011 23:10
>> On 2011-09-13 06:30, Lothar Werzinger wrote:
>> > What is the trick to build and create p2 sites that work like the default
>> http://download.eclipse.org/tools/buckminster/headless-3.6 and don't produce errors on install?
>> >
>> Use Buckminster-3.7 at all times.
>>
>> - thomas
>
>
> The hudson buckminster plugin only supports 3.5 and 3.6. That and the latest fix you made were why I decided to build it
> myself, but as outlined I had problems with the created p2 sites.

That's not true. We use both Jenkins and Hudson with Buckminster 3.7.

- thomas

Maybe (*), but that's not my problem here, as the released 3.7 won't have your fix int it, and the eclipse automated build seems not to produce any consumable p2 sites.

What I need is either a p2 site from the eclipse automated build from GIT where I can install from (to get the latest fix) or some advice how to build buckminster myself in a way that it will be installable without error. That's what I asked for in my previous post.

*)
The hudson plugin currently has only options to install 3.5 and 3.6 and the author has confirmed that in an email. In order to install my built version I had to hand edit some JSON file (was attached in a previous post). I assume I could do a similar edit for 3.7, but that would still not get the the fix I am eager to test.

On 2011-09-13 16:44, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Tue, 13 September 2011 07:11
>> On 2011-09-13 16:02, Lothar Werzinger wrote:
>>
>> > Maybe (*), but that's not my problem here, as the released 3.7 won't have your
>> > https://bugs.eclipse.org/bugs/show_bug.cgi?id=357060 int it
>>
>> Yes it has.
>>
>> - thomas
>
>
> Oh, I was not aware of that. I was under the impression that it contains a stable release that does not change that
> frequently. I will try it.
> Btw. does http://download.eclipse.org/tools/buckminster/headless-3.7
> contains always the latest from git or just selected backported fixes?
>
> It would still be nice to know why the built buckminster from git did not install, though.
>
Hard to say. One difference against our own build is that you never built the workspace properly. We use the command
sequence:
import
build
perform

Also, the build.properties from the org.eclipse.buckminster.releng project should be fed to each build step with the -P
option. You can look at the 'promote.bmscript' file in the releng project as a reference.

On 2011-09-13 16:44, Lothar Werzinger wrote:
> It would still be nice to know why the built buckminster from git did not install, though.
>
Hard to say. One difference against our own build is that you never built the workspace properly. We use the command
sequence:
import
build
perform

Also, the build.properties from the org.eclipse.buckminster.releng project should be fed to each build step with the -P
option. You can look at the 'promote.bmscript' file in the releng project as a reference.

- thomas

Thanks, I'll try.

Maybe it would be helpful if you could publish a working hudson config.xml that builds p2 site(s) on the wiki for those who want or need to build themselves.

On 2011-09-12 22:06, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Mon, 12 September 2011 05:19
>> Hi Lothar,
>>
>> I found the cause of this problem and fixed it. You should get rid of the problem if you update to the most recent
>> Buckminster. More info here:
>>
>> Bug 357060 - Generated conflicting source dependencies are sometimes left out
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=357060
>>
>> Regards,
>> Thomas Hallgren
>
>
> That's great news. Will there be a new release?

This fix has already been pushed to our update sites. So as I wrote earlier, you should get rid of the problem by just
updating to the most recent Buckminster.

On 2011-09-13 18:54, Lothar Werzinger wrote:
> I just tried with http://download.eclipse.org/tools/buckminster/headless-3.7, but I still don't get org.apache.commons.logging.source 1.1.1 in my TP.
>
> No component named org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721] is known to Buckminster
>
> Full console log attached.

That's probably because you don't have the orbit update site enabled in your rmap. That's the only place where that
source can be found.

On 2011-09-13 18:54, Lothar Werzinger wrote:
> I just tried with http://download.eclipse.org/tools/buckminster/headless-3.7, but I still don't get org.apache.commons.logging.source 1.1.1 in my TP.
>
> No component named org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721] is known to Buckminster
>
> Full console log attached.

That's probably because you don't have the orbit update site enabled in your rmap. That's the only place where that
source can be found.

- thomas

Ahh, strange the 1.0.4 source bundle is available, as is the 1.1.1 binary bundle.
I'll try to add orbit and see how that goes.

On 2011-09-13 22:23, Lothar Werzinger wrote:
> This whole thing seems to become the never ending story :(
>
> I added the http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/ but I still get the
>
> No component named org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721] is known to Buckminster
>
That's very odd. I just tried your cquery/rmap with orbit enabled, and I get that source included.

> I also wonder if the Artifact repository out of sync messages are harmful and why they get generated.
>
They generally mean that the artifacts.xml lacks some jars that apparently are present. This typically happens when a
materialization is interrupted. The cure is to simply clear the TP completely and start over.

On 2011-09-13 22:23, Lothar Werzinger wrote:
> This whole thing seems to become the never ending story
>
> I added the http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/ but I still get the
>
> No component named org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721] is known to Buckminster
>
That's very odd. I just tried your cquery/rmap with orbit enabled, and I get that source included.

Hmm, not here:

org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721](&(buckminster.download.source=true)(!(eclipse.p2.optional=false))): Rejecting provider p2(http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/?importType=binary[http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/?importType=binary]): No component match was found
org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721](&(buckminster.download.source=true)(!(eclipse.p2.optional=false))): No provider was found that could resolve the request

What Orbit update site are you using? Could you attach the rmap you are using?

Thomas Hallgren wrote on Tue, 13 September 2011 13:46

> I also wonder if the Artifact repository out of sync messages are harmful and why they get generated.
>
They generally mean that the artifacts.xml lacks some jars that apparently are present. This typically happens when a
materialization is interrupted. The cure is to simply clear the TP completely and start over.

- thomas

Strange, my build always starts with an empty TP

WARN: Target platform directory 'bm-workspace/tp-full/' does not exist and will be created

You can remove the ?importType=binary part. That's an old relic for the now deprectated 'eclipse.import' reader type. It
has no meaning for the 'p2' reader.

- thomas

On 2011-09-13 23:51, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Tue, 13 September 2011 13:46
>> On 2011-09-13 22:23, Lothar Werzinger wrote:
>> > This whole thing seems to become the never ending story :(
>> >
>> > I added the http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/ but I still get the
>> >
>> > No component named org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721] is known
>> to Buckminster
>> >
>> That's very odd. I just tried your cquery/rmap with orbit enabled, and I get that source included.
>
>
> Hmm, not here:
>
> org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721](&(buckminster.download.source=true)(!(eclipse.p2.optional=false))):
> Rejecting provider
> p2(http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/?importType=binary[http://download.eclipse.org/tools/orbit/downloads/drops/R20110523182458/?importType=binary]):
> No component match was found
> org.apache.commons.logging.source:osgi.bundle/[1.1.1.v201101211721,1.1.1.v201101211721](&(buckminster.download.source=true)(!(eclipse.p2.optional=false))):
> No provider was found that could resolve the request
>
>
> What Orbit update site are you using? Could you attach the rmap you are using?
>
> Thomas Hallgren wrote on Tue, 13 September 2011 13:46
>> > I also wonder if the Artifact repository out of sync messages are harmful and why they get generated.
>> >
>> They generally mean that the artifacts.xml lacks some jars that apparently are present. This typically happens when a
>> materialization is interrupted. The cure is to simply clear the TP completely and start over.
>>
>> - thomas
>
>
> Strange, my build always starts with an empty TP
> WARN: Target platform directory 'bm-workspace/tp-full/' does not exist and will be created