Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team; the main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

+

Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team. (See [[NetBeansArchivedFeatures]] for an overview.)

+

+

The main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

Line 76:

Line 78:

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

+

+

Can also check for historical modules since deleted:

+

+

main$ (cd .hg/store/data; ls -1d *ruby*)

+

+

which turns up just one additional item, _ruby_application1, which was bogus and need not be converted.

+

+

Should also check for related modules in contrib:

+

+

contrib$ ls -1d *ruby*

+

ls: cannot access *ruby*: No such file or directory

+

+

but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.

+

+

Can also check for related files in misc. In this case you may need to use the rename directive to convert, since file layout here is arbitrary.

Also removing nb.cluster.ruby from clusters.config.bloated.list, and removing ruby.themes from experimental cluster. Can also find some other mentions of the Ruby support that need to be cleaned up this way:

+

+

main$ hg locate -r . -0 nbbuild\*\* \\.\* | xargs -0 grep -l -- ruby

+

+

23c261cc6b0b is the resulting commit; 99934ed62e8e is the merge also reflecting the removal of the module sources.

Using IDE, created a new module suite in a temp dir with the name 'ruby', against a platform named 'nb69'. Closed project. From command line, created nbproject dir in new repo, and moved three files into it from template project:

-

==Alternate: converting to Maven reactor tree==

+

nbproject/project.xml

+

nbproject/project.properties

+

nbproject/platform.properties

-

XXX

+

Opened the root of the new repo as a project, which is now a suite; this generates some more files:

+

+

build.xml

+

nbproject/build-impl.xml

+

genfiles.properties

+

+

plus some nonversionable stuff in nbproject/private. Right-click project, Rename, type in 'NB Ruby' (do not move!); this just sets app.title in project.properties for display purposes. (Want to keep <name>ruby</name> in project.xml for use as cluster name.) Also in platform.properties, unselect all clusters from platform except

+

+

platform

+

harness

+

ide

+

nb

+

+

Can soon Add Existing... to add the modules beneath it, but this is not available for converting former netbeans.org modules until they are prepared a bit better; in the current state they are not even loadable as projects. First need to insert <standalone/> in each project.xml so they are treated as self-contained module projects:

+

+

ruby$ perl -pi -e 's! <module-dependencies! \

+

<standalone/>\n <module-dependencies!' */nbproject/project.xml

+

+

Now Add Existing... works. Check that we did all of them:

+

+

ruby$ fgrep standalone */nbproject/project.xml

+

+

Need to update our ignore file so we can see what we really need to add. We append to the imported entries from main to exclude build and nbproject/private subdirs of every project (suite & modules):

+

+

^o.jruby.distro/unpatched_source$

+

^o.jruby/patched_source$

+

(^|/)build$

+

(^|/)nbproject/private$

+

+

Now we can track all the new files:

+

+

ruby$ hg add

+

+

There is still a problem that every module's build.xml tries to refer to ../nbbuild/templates/projectized.xml, which is for netbeans.org modules only. We could have solved this simply by deleting build.xml before "Add Existing", causing them to regenerated with the correct import, but a lot of these build scripts have customizations we need to keep. So instead:

+

+

ruby$ perl -pi -e 's!\.\./nbbuild/templates/projectized\.xml!\

+

nbproject/build-impl.xml!' */build.xml

+

+

There's a lot more cleanup to do, but let's commit what we have so far: c8c08c1453fc

+

+

==Handling external binaries==

+

+

We will also handle */external/binaries-list files as [[ExternalBinaries]] describes: e54ce82201a2

+

+

That just made sure the binaries are downloaded, which is required for in-IDE code completion and so on. The original projects also used release.* properties to copy JARs to $cluster/modules/ext as [[DevFaqExternalLibraries#NBM_build.2C_managing_correct_license_for_NBMs]] describes; this not available for external projects in the 6.9 harness, so we need do some more work: e69851c445db

+

+

==Fixing up suite build==

+

+

Time to try our first suite build. We have to make some corrections for different Ant target names in overrides, and make sure that external binaries are downloaded at the start of the build: 57a65875b2c5

+

+

Trying again, we run into a real problem:

+

+

ruby.build-init:

+

Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-csl-api.jar

+

because of dependency: org.netbeans.modules.csl.api/2 > 2.11

+

+

Seems that there was some sort of incompatible change in the CSL API in 7.0, whereas for now we are building against 6.9. Inspection of the changeset bd130b29325c as it applied to the ruby cluster shows that no code in the Ruby support actually changed, meaning the change was probably bogus (original author should have merely pushed a fixed version of editor lib without touching downstream modules!) and we are safe just reverting the spec version in the dep: d08f4934b8b6

+

+

The next problem:

+

+

ruby.build-init:

+

Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor.jar

+

because of dependency: org.netbeans.modules.editor/3 > 1.53

+

+

Again inspection of the changeset 10f8e107bf98 shows an increment in module deps for no clear reason; we will just back it out: deace5ee3cf6

+

+

Now we have:

+

+

ruby.build-init:

+

Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor-lib.jar

+

because of dependency: org.netbeans.modules.editor.lib/3 > 3.1

+

+

Inspection of b6f862b9772e shows that the author made no code changes in the ruby cluster yet requested a new major version. This is simply wrong; he should have used a release version range instead. Again we will fix this: 79e7ae432654

+

+

Next is an error in RubyCodeCompleter; da32d187895f consumed an incompatible API change without proper versioning or a compatibility layer. Looks like we can work around this: a42810a50c00

+

+

Now it seems ruby.merbproject is uncompilable - not surprising given that it was not being built continuously. We will just remove the three "experimental" modules from the suite for now: 87b8450b7476

+

+

Next:

+

+

ruby.testrunner.build-init:

+

Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-gsf-testrunner.jar

+

because of dependency: org.netbeans.modules.gsf.testrunner > 1.16

+

+

This is more serious because main #f3f2a4970c8d was clearly incompatible (changed signature of an interface). We can make it compilable against 6.9 and time will tell if the result links against 7.0: 30de9b8f9784

+

+

Finally the modules all compile!

+

+

Building NBMs fails due to

+

+

spellchecker.bindings.ruby.nbm:

+

.../nb69/harness/suite.xml:222: The following error occurred while executing this line:

which can be fixed by removing a bogus definition of license.file: 7a8ad2a1e64f

+

+

==Fixing tests==

+

+

Unit tests are still uncompilable since CslTestBase cannot be found (Ant-based NB modules cannot publicly export test utilities); we need to copy that into ruby.platform test sources from 6.9 sources, along with various other test utility dependencies: d12ce02008b8

+

+

Now all error badges are gone, but some tests can still not be built, much less run:

+

+

ruby.platform.test-unit-dep-build:

+

.../nb69/harness/common.xml:524: The following error occurred while executing this line:

To make it possible for all developers to build against a controlled copy of NetBeans, we will download it right away in the suite build. We put this in the .hg subdir so that hg clean (including from Hudson) will not remove it: a292d486d861

+

+

http://source.apidesign.org/hg/netbinox/file/1075e21a049d/platform.xml demonstrates another possible approach which is more flexible in that it lets you pick up just the clusters you need. The download is potentially slower in this case since each module has to be retrieved separately.

+

+

=Alternate: converting to Maven build=

+

+

Not (yet) done in this case so cannot give exact steps. The basics would be:

# Create Maven POMs. One pom-packaging project at root with <modules>. One nbm-packaging project per module. And one nbm-application-packaging project to represent the "suite", with dependencies on the modules.

+

# Remove all build.xml. Replace any custom build functionality not handled by Maven by default with the appropriate Maven plugins.

Introduction

Some module clusters in the NetBeans main repository are no longer actively developed or supported by the NetBeans team. (See NetBeansArchivedFeatures for an overview.)

The main repo is intended only for stable, actively developed modules. To make it feasible for volunteers to take over maintenance of the "downsized" modules, the modules need to be moved into their own source repositories, converted to be buildable in a standalone fashion against NetBeans platform & core IDE binaries (modules in main and contrib use a special legacy build convention), and otherwise cleaned up.

This page lists the steps I performed while running this process for the NetBeans Ruby support.

Moving Source Code

Identifying modules to be moved

First it is necessary to decide exactly which modules (~ top-level directories) need to be moved. Open nbbuild/cluster.properties and look for the cluster definition; these are all the modules normally built in the cluster.

Also look around for other modules in the source tree which appear to be related, though they may not currently be built in that cluster, perhaps because they are poorly maintained. They should still be moved; the new developers can decide to fix them up and include them in builds, delete them, leave them in sources but unbuilt, etc. For example, searching by name:

In this case you might have noticed the libs.bytelist and libs.jvyaml modules. But these are used by the languages.yaml module, which despite its Ruby associations is in the ide cluster (a dep of editor.kit): YAML text file support is for now at least a standard part of the base IDE distribution, since it is occasionally used in non-Ruby-based app frameworks.

Can also check for historical modules since deleted:

main$ (cd .hg/store/data; ls -1d *ruby*)

which turns up just one additional item, _ruby_application1, which was bogus and need not be converted.

but there appear to be none. If there were, these would likely need to be converted as well; removed from contrib; and the conversion result forcibly merged into the conversion result from main (so as to provide both sets of modules). As with modules in main not in the official cluster, these additional modules might or not not be used, but it would be best to make them available just in case.

Can also check for related files in misc. In this case you may need to use the rename directive to convert, since file layout here is arbitrary.

Now for the actual conversion. Currently Mercurial releases are incapable of doing this in a reasonable amount of time for a repo as big as main, due to http://mercurial.selenic.com/bts/issue991, so we need to use a patched version.

For simplicity, we will assume that the converted modules should live in the same source structure as they do in main, so we can use the include directive but no rename directive in the convert command.

In order to capture historical branch information - i.e. changes made to IDE release stabilization branches to Ruby modules, usually backports of fixes from "trunk" - I will actually switch to another local repo which contains everything in http://hg.netbeans.org/core-main/ and http://hg.netbeans.org/releases/ too. This is optional but might be useful. Say a Ruby developer receives a bug report with a stack trace from NetBeans 7.0 beta 1 - they might like to check out the tip of the release70_beta branch to make sure line numbers in their sources exactly match the stack trace.

This will do a few preparatory calculations, then start counting down changesets in historical order as it searches for relevant file modifications and converts them. Note that scanning gets slower per changeset as you get into newer history. You can watch progress from another shell window:

Also removing nb.cluster.ruby from clusters.config.bloated.list, and removing ruby.themes from experimental cluster. Can also find some other mentions of the Ruby support that need to be cleaned up this way:

main$ hg locate -r . -0 nbbuild\*\* \\.\* | xargs -0 grep -l -- ruby

23c261cc6b0b is the resulting commit; 99934ed62e8e is the merge also reflecting the removal of the module sources.

Publishing separate repo

Preparing Standalone Build

Converting to Ant-based module suite

Using IDE, created a new module suite in a temp dir with the name 'ruby', against a platform named 'nb69'. Closed project. From command line, created nbproject dir in new repo, and moved three files into it from template project:

Opened the root of the new repo as a project, which is now a suite; this generates some more files:

build.xml
nbproject/build-impl.xml
genfiles.properties

plus some nonversionable stuff in nbproject/private. Right-click project, Rename, type in 'NB Ruby' (do not move!); this just sets app.title in project.properties for display purposes. (Want to keep <name>ruby</name> in project.xml for use as cluster name.) Also in platform.properties, unselect all clusters from platform except

platform
harness
ide
nb

Can soon Add Existing... to add the modules beneath it, but this is not available for converting former netbeans.org modules until they are prepared a bit better; in the current state they are not even loadable as projects. First need to insert <standalone/> in each project.xml so they are treated as self-contained module projects:

There is still a problem that every module's build.xml tries to refer to ../nbbuild/templates/projectized.xml, which is for netbeans.org modules only. We could have solved this simply by deleting build.xml before "Add Existing", causing them to regenerated with the correct import, but a lot of these build scripts have customizations we need to keep. So instead:

There's a lot more cleanup to do, but let's commit what we have so far: c8c08c1453fc

Handling external binaries

We will also handle */external/binaries-list files as ExternalBinaries describes: e54ce82201a2

That just made sure the binaries are downloaded, which is required for in-IDE code completion and so on. The original projects also used release.* properties to copy JARs to $cluster/modules/ext as DevFaqExternalLibraries#NBM_build.2C_managing_correct_license_for_NBMs describes; this not available for external projects in the 6.9 harness, so we need do some more work: e69851c445db

Fixing up suite build

Time to try our first suite build. We have to make some corrections for different Ant target names in overrides, and make sure that external binaries are downloaded at the start of the build: 57a65875b2c5

Trying again, we run into a real problem:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-csl-api.jar
because of dependency: org.netbeans.modules.csl.api/2 > 2.11

Seems that there was some sort of incompatible change in the CSL API in 7.0, whereas for now we are building against 6.9. Inspection of the changeset bd130b29325c as it applied to the ruby cluster shows that no code in the Ruby support actually changed, meaning the change was probably bogus (original author should have merely pushed a fixed version of editor lib without touching downstream modules!) and we are safe just reverting the spec version in the dep: d08f4934b8b6

The next problem:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor.jar
because of dependency: org.netbeans.modules.editor/3 > 1.53

Again inspection of the changeset 10f8e107bf98 shows an increment in module deps for no clear reason; we will just back it out: deace5ee3cf6

Now we have:

ruby.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-editor-lib.jar
because of dependency: org.netbeans.modules.editor.lib/3 > 3.1

Inspection of b6f862b9772e shows that the author made no code changes in the ruby cluster yet requested a new major version. This is simply wrong; he should have used a release version range instead. Again we will fix this: 79e7ae432654

Next is an error in RubyCodeCompleter; da32d187895f consumed an incompatible API change without proper versioning or a compatibility layer. Looks like we can work around this: a42810a50c00

Now it seems ruby.merbproject is uncompilable - not surprising given that it was not being built continuously. We will just remove the three "experimental" modules from the suite for now: 87b8450b7476

Next:

ruby.testrunner.build-init:
Cannot compile against a module: .../nb69/ide/modules/org-netbeans-modules-gsf-testrunner.jar
because of dependency: org.netbeans.modules.gsf.testrunner > 1.16

This is more serious because main #f3f2a4970c8d was clearly incompatible (changed signature of an interface). We can make it compilable against 6.9 and time will tell if the result links against 7.0: 30de9b8f9784

Finally the modules all compile!

Building NBMs fails due to

spellchecker.bindings.ruby.nbm:
.../nb69/harness/suite.xml:222: The following error occurred while executing this line:
.../nb69/harness/common.xml:426: Exception reading blurb from .../ruby/standard-nbm-license.txt

which can be fixed by removing a bogus definition of license.file: 7a8ad2a1e64f

Fixing tests

Unit tests are still uncompilable since CslTestBase cannot be found (Ant-based NB modules cannot publicly export test utilities); we need to copy that into ruby.platform test sources from 6.9 sources, along with various other test utility dependencies: d12ce02008b8

Now all error badges are gone, but some tests can still not be built, much less run:

ruby.platform.test-unit-dep-build:
.../nb69/harness/common.xml:524: The following error occurred while executing this line:
.../nb69/harness/common.xml:519: Cannot open .../nb69/platform/lib/nbproject/project.xml

which seems to be a reporting bug in the harness; the problem is <test/> on platform modules remaining in ruby.platform/nbproject/project.xml, which needs to be removed.

Inspection of project.properties also shows that it is still using old-style test deps:

Locating platform binaries

To make it possible for all developers to build against a controlled copy of NetBeans, we will download it right away in the suite build. We put this in the .hg subdir so that hg clean (including from Hudson) will not remove it: a292d486d861

Alternate: converting to Maven build

Not (yet) done in this case so cannot give exact steps. The basics would be:

Remove all nbproject dirs.

Create Maven POMs. One pom-packaging project at root with <modules>. One nbm-packaging project per module. And one nbm-application-packaging project to represent the "suite", with dependencies on the modules.

Remove all build.xml. Replace any custom build functionality not handled by Maven by default with the appropriate Maven plugins.