Contents

A recap of the situation

Today we tend to have a source feature for every binary feature. The source features include a source plugin as well as some number of ISV doc and potentially example plugins.

The source plugin in turns contains an "src" directory and in that directory is a directory for every plugin in the corresponding binary feature. Each source folder contains one or more source zips corresponding to the JARs in the binary versions of those plugins. The plugin also contains a plugin.xml that contributes an extension to the org.eclispe.pde.core.source extension point. An example is given below.

Today the Eclipse project ships 4 source features for a total of 6 source plugins (there are some platform-specific source plugins). Each of the plugins is installed as a directory rather than being a JAR.

The problem

The 3.3 plan includes an item for shipping finer grained components. This is meant to make acquiring Help and Update function (for example) easier as there would be a well-defined feature that captures the set of bundles needed for that function.

If we follow the current pattern, each of these new features will spawn a new source feature. These features and plugins are autogenerated so this is not an overwhelming development load but it does increase the feature count significantly. Note also that there will be a large number of features containing only one (or a small number) of plugins. Function such as Search, Jobs, etc can quite reasonably be used independently. The point of the 3.3 plan item is to make these easily accessible.

The Orbit project will exacerbate this problem by introducing a large number of standalone bundles for which we will want to make source available. Creating binary and source features for each of these would not be feasible. Note also that in many cases the Orbit team will have source archives from the originating project and would like to avoid repackaging them. Instead the should be able to bundle them as they would any other JAR.

There is an additional problem discovering where the source for a given plugin lives. Take this little quiz: Where is the source for org.eclipse.help? And for org.eclipse.help.base? The challenge is in mapping back from binary bundle to containing feature and then to source feature. The answers to the quiz are not given here. The best way to understand the problem is to go through the pain of this process yourself, so this is left as an exercise for the reader.

The solution

The proposal here is to package the source for each bundle in a corresponding source bundle. The 3.3 plan also includes some work towards finer-grained (well righter-grained) provisioning. This will enable delivery of the new finer grained components and allow these new source bundles to be provisioned independently (or at least at the right granularity). For example, developers would not need to download all of the source for the Platform just to see the source for one part.

Note that this eliminates the need for reverse mappings from binary to source. The source bundle id is either a mechanical mapping of the binary bundle id or explicitly noted in the metadata of the binary bundle.

One of the first reactions is "that will double the number of bundles in the SDK and performance will take a nose dive". True and false. If we ship all the source bundles in the SDK then indeed the bundle count will double. However, there is actually no need whatsoever for the source bundles to be "installed". That is, the IDE itself does not make use of the bundles. The source developers want is the source of the bundles in the __target__. By default the running IDE configuration is the target and the source plug-ins are there. We just don't need to install them in the runtime.

As pointed out before, as we advance in the provisioning story, it will become increasingly irrelevant to ship all the source with the SDK. Developers will be able to get the source quickly and easily as they would any other bundle.

Outstanding issues

The shape of the source contribution needs to be ironed out. The extension mechanism should be retained for backwards compatibility but the new mechanism likely should avoid injecting plugin.xmls if possible. Perhaps some markup in the source bundle's manifest.mf

The update.configurator and other mechanisms may need to be modified to avoid installing source bundles into the Equinox framework.

PDE Build needs to generate the new bundles and the source feature generator would have to be updated to do the new thing. This may be subtle if we want to be able to generate source features in the old form.

Implementation

Source bundles were included in the SDK as of the N20071128-0010 build. See Andrew's note to [eclipse-dev] with links to the relevent bugs that were needed to implement this functionality.

To incorporate source bundles into your build you'll need to

Switch to a shiny new version of pde build that includes support for generating source bundles. Please use N20071128-0010 build or later vintage.

Update the builder for your top level feature with individualSourceBundles=true. During the SDK build, we generate the build scripts for the source build, and then build a master feature that includes all the remaining features in the build that are signed at eclipse.org. Thus, to transition to generating source bundles, we updated the builder's build.properties of the sdk and master feature to specify individualSourceBundles=true. If you build all your features individually, you'll need to update the builder's build.properties for each. See bug bug 208821 for more details.

bug 208816 for an example of the required changes. Note that this example is different than most cases in that we store swt source in the source fragment, and we don't generate a org.eclipse.swt.source bundle.

Set a flag to not generate a source bundle if the bundle doesn't include source. Set generateSourceBundle=false in the build.properties of that bundle. Bug 210456 provides an example.

Exclude source bundles in generated source features where the plugin doesn't contain source. For instance, all our org.eclipse.*.doc.user plugins don't include source, so these are excluded during the generation of the source features. For example, see bug 208821 comment #5 for more details.