The Problem

Writing NetBeans module code frequently involves registering Java objects in the system.
Traditionally this has required editing of metadata files.
For example, to add a menu item you need to add an entry to your layer.xml
(or perhaps multiple entries).
To add a (global) query implementation you need to create a META-INF/services/... file.

Dealing with these metadata files in addition to regular Java code poses several problems:

Figuring out what metadata file you need to create,
or entry you need to add to an existing metadata file,
is generally not obvious, even to seasoned module developers.
You need to consult external HTML documentation, rely on wizards to do it for you,
or look around at blogs and sample code to find an example.
Even adding a relatively simple global singleton service often stumps new Platform users.

There is no build-time syntax checking in metadata files.
A file META-INF/service/some.Interface will be silently ignored (missing s!).
The syntax for the name and attributes of *.instance files is subtle
and rarely understood in full.
A mistyped class or method name will result in a runtime exception which must be traced back to the original source code error.
XML layer files can get rather long, and subtle syntactic mistakes are difficult to spot.

When reading source code, it is not always obvious that some class is being registered as an object.
You may innocently add a parameter to a constructor,
not realizing until runtime that it needed to have zero arguments in order to be loaded.
People sometimes delete a metadata registration but forget to delete the class
(or method) it was registering.

The module development support GUI needs to make special effort to register metadata in addition to classes.
There is no such thing as a single-file template to add a menu item, for example.
Similarly, published example code has to display several different files to show how to implement one logical action.

The module development support also needs special refactoring hooks to update metadata,
which are incompletely implemented.

The Proposed Solution

The solution is to use JSR 269 annotation processors.
There are two uses.

Generic service annotation

Platform infrastructure will define a generic @Service annotation
which can register services to global lookup.
Any SPI which requires instances of objects to be added to global lookup
can immediately be used with this annotation.

is all of the code necessary to register a custom MIME resolver into global lookup.

The annotation has some optional parameters, e.g. to let you order services.
Interestingly, it also has explicit support for META-INF/namedservices, or "named lookup";
any SPIs which use Lookups.forPath will support registrations made using this annotation.
However, it is expected that such registrations would normally use custom annotations.

Layer-generating annotations

Platform experts who define an SPI that requires declarative registrations
in the system filesystem (~ XML layer)
can also define Java annotations for these registrations,
and fairly easily add annotation processors to transform them into layer fragments.

could suffice to register a hyperlink provider for Java source files.
Internally this would generate the same layer entry as you formerly needed to make by hand,
though into a file build/classes/META-INF/generated-layer.xml.

It is important to note that this all happens in the Java editor
and can take advantage of regular Java editing features:

The registration annotation can be @linked to from the Javadoc of the interface.

Code completion offers the annotation and its parameters.

If NetBeans supported JSR 269, then the processor could also supply code completion on annotation parameter values, helping to offload some of the functionality of apisupport.project wizards into the actual API where it belongs.

Find Usages on the annotation can be used to jump to all registrations in open projects.

Renaming or moving the annotated class or method does not break registration.

Any Java IDE will catch syntax errors such as a mistyped annotation name. 269-compliant IDEs will catch semantic errors too, such as an attempt to annotate a class not implementing the right interface. (NetBeans will currently report semantic errors during the Ant build but not yet in the editor window.)

One annoyance is that the Java editor will mark private static factory methods as "unused"
(underlined in a gray wavy line) even when they are annotated to be registered.
This could only be solved using a
meta-annotation.
Still, the situation is better than when using explicit layer registrations,
when there is no indication in the source code that the factory method is used for anything at all,
and there is a real danger of it being casually deleted.

Non-object-oriented annotations

It is an open question how many layer entries should be created via annotations.
When the entry is clearly registering a Java object in the system,
annotations are definitely recommended.
For example, if you wanted to register a custom New File wizard,
the following might be sensible:

However NetBeans also supports New File wizard entries which are just plain text files,
optionally with some FreeMarker template substitutions but requiring no Java code.
Can and should such entries be registered using annotations?

As another example, JavaHelp help sets are registered in the platform
by bundling the actual help set data files,
and then adding a one-line XML file to Services/JavaHelp in the layer.
There is no associated Java class to attach an annotation to.
(Technically you could add a custom javax.help.HelpSet object to global lookup,
but in practice this kind of dynamic behavior is rarely if ever needed.)

If it is desirable to support annotations dissociated from Java instances,
it should be possible by defining them on package-info.java in the module:

Note that it is possible for a processor to verify the existence and even syntax
of an external resource referenced by such an annotation at compile time.
(This will not work in the case of @HelpSetRegistration above,
since the javahelp/ directory is not part of the source path.)

Problems with package annotations

There are some complications with such a proposal:

Only one annotation of a given type can be registered on a single package element,
so if multiple registrations are needed,
the annotation definition would need to explicitly accommodate this
e.g. using arrays of nested annotations.

Different tools invoking javac (e.g. Ant) may have inconsistent policies
as to when to pass package-info.java to the compiler during an incremental build,
since no package-info.class is generated
unless some annotation with class or runtime retention is present.
Inconsistencies could result in registrations, or changes to registrations,
being omitted under some development workflows.
In particular, under Ant 1.7.1, a package annotation with source retention
will be "recompiled" after changes only when you do a clean build of the module;
even if you use class retention, an interaction with <depend>
means that changes are processed only after two incremental builds.
This is fixed in Ant 1.8.0.

The advantage to documentation and module development templates of a registration
being self-contained does not apply in this case:
you just need to add one annotation to a Java source,
but this is in addition to the resource file you are registering,
and you may need to edit an existing package-info.java.

A module will usually contain more than one package,
and there is not always one obvious "main" package.
Since any of them could be annotated interchangeably,
confusion could arise from having multiple annotated package-info.javas in one module.

Folders

A related question is what to do about folders.
In certain cases, the position, display name, or other attributes of a folder in the SFS are important. Examples include:

menus (and certain submenus) in the main menu bar

toolbars (perhaps; there is also an XML file which partially takes over)

New File or New Project wizard categories

project Properties dialog categories

Such folders are not associated with any particular instance inside them.
Should non-object annotations be used to define them?

(Even without annotations, such folders are problematic because it is far from clear
which module ought to be defining the folder's properties, when several modules add entries.
Currently we work around this ambiguity by making ValidateLayerConsistencyTest fail
in case more than one module declares attributes for the same folder
yet they do not agree on the values.
This trick does nothing to make the platform easier to use,
and in fact only works for modules in the standard IDE distribution.)

Named service registrations

@ServiceProvider supports META-INF/namedservices lookup (i.e. Lookups.forPath).
This system is attractive because the runtime overhead of loading a tiny text file
is likely less than that of parsing and merging an XML file,
creating a chain of FileObjects,
translating them into DataFolders and InstanceDataObjects,
and running a FolderLookup over them all for their InstanceCookies.
However, it is not very pleasant to register named instances using this annotation.
After all, you need to type in a path - a bit of metadata not verified statically -
retaining some of the disadvantages of explicit layer registration.

Perhaps certain annotations could register into this directory rather than a layer.
This would apply only to object-oriented registrations for which a path, object,
and perhaps optional position
comprise all the relevant information.
(I.e. there must be no semantic significance to
the display name of the instance,
other ad-hoc file attributes,
labels or attributes on containing folders,
etc.)
Many, perhaps most, layer registrations would fall into this category.

For example, a file
META-INF/namedservices/Projects/org-netbeans-modules-java-j2seproject/Nodes/org.netbeans.spi.project.ui.support.NodeFactory
could specify

Problems with named registrations

The corresponding SPI needs to use Lookups.forPath
rather than directly searching the filesystem.
In most cases where this is needed it could be done easily.
The MIME Lookup API may need to be retrofitted.
TBD whether Lookups.forPath should continue to look in recursive children
of SFS folders for instances
(this is not done for META-INF/namedservices
and is probably unwanted by most SPI callers).

Services lookup does not currently support factory methods,
so only a public class with a default constructor could be registered in this way.
Either services lookup could be enhanced to support factory methods
(another departure from the Java standard,
and @ServiceProvider would need to support this too);
or the processor would silently switch to layer registration when given a factory method.

The current annotation processor infrastructure does not make it safe
for processors other than that for @ServiceProvider
to write resources into META-INF/namedservices.
(JSR 269 forbids a resource to be written more than once per job.)
As for META-INF/generated-layer.xml,
different processors would need to cooperate on writing these resources.

Major open issues

Non-object-oriented registration

The main open issue is what to do with non-object-oriented registrations,
as discussed above.

Related to this is the question of how to deal with things like menu folders and separators.

Named-service generating annotations

Can and should this be used?

Transition plan

When introducing a new annotation,
the developer of the annotation and processor
has the responsibility to ensure it is used in old code.
Currently there is no infrastructure to do this short of text search-and-replace.

Some kind of generic script may be needed
to search through existing source XML layers
for certain patterns and replace them with source annotations.
Such a script would need to deal with a lot of corner cases,
such as an object being registered from a different module
than the one which contains the Java code
(in such a case the registration cannot be converted to an annotation).

Naming conventions

If annotations are used widely, there will be a lot of different people defining them.
We need some conventions for how annotations and their attributes should be named.

Throughout this document I am assuming that an interface named Something
will normally be registered using a nested annotation named Something.Registration.
This is simple to remember and has the advantage that when browsing
Javadoc, code completion, or sources, you are likely to see the annotations easily.

Some conventions are needed for annotation attribute naming.
position is obvious enough,
but service, projectType, etc. also need to be consistent.

<XML layer in context>

Currently does not show entries that would be generated by annotations
in source modules.
(Shows all entries from the binary platform already.)
Could be fixed in various ways if considered important enough.

Testing support

You can already easily test that generated resources are correct;
just use the annotation on a dummy class in unit tests.

To test that the processor rejects various mistakes
(or to test that warning/error messages are correct)
you would need to run JSR 199 from the unit test.
There is no standardized support for this yet.

Alternatives Considered

Eclipse-style metadata files

Eclipse defines "plug-in manifests", which are XML files listing declarative registrations
made by the plug-in to predefined "extension points".
Syntaxes of individual sections of the manifest are specified by XML schemas
bundled with the SPI plug-in.
Since the IDE provides a schema-directed editor,
and the plug-in build process validates the manifest,
syntax errors are caught early and well reported.

This system works reasonably well for registrations not involving Java objects,
e.g. help system fragments.
When tied to Java objects, however, this system suffers from many of the same drawbacks
as the NetBeans layer registration, being improved only in syntax checking and discoverability.

Runtime layer generation

Rather than statically creating the layer fragment from the annotation during the build,
the build could create a simple index of relevant annotations in a module,
and the module system could dynamically insert the corresponding SFS fragment when the module is loaded.

This scheme would have some minor advantages:

Fixes to the logic of annotation-to-layer conversion would take effect even on modules built in the past, whereas static generation can result in "stale" registrations comparable to inlining old compile-time constants.

Fragment conversion could make use of some "live" information in the platform. For example, FileObject attributes could be live Java objects which would not be easily represented in serialized form.

Disadvantages are several:

An index of annotations is still required for efficiency, so implementation is more complex.

Computing the SFS fragment could add overhead to startup, especially if caches are thereby invalidated.

It is hard for a module developer to see what SFS fragment is being generated, which complicates debugging.

Custom annotation indexing

A more radical approach would be to use a system which bypasses the SFS entirely.
The motivation for doing so is that SFS registration of objects is a quite complex system,
involving multiple layered APIs (Filesystems, Datasystems, Lookup, "Settings", ...).
Each layer of code adds inherent overhead and points of failure,
which makes dealing with settings and configuration in the NB platform
very hard to understand at a deep level, tricky to optimize,
and prone to all manner of threading bugs and corner cases.

A framework such as SezPoz
could cover the basic requirements for registration in a large platform like NetBeans,
with a far simpler infrastructure:
modular, declarative registration of services, simple syntax verified at compile time,
with service metadata inspectable prior to class loading.

Unfortunately, there are two main blockers for such a transition:

NetBeans' SFS registration provides some special features which would need to be replicated in any other system:

Fine-grained service change notification, e.g. upon module installation into a running platform.

A generic format for branding (or masking) registrations made in other modules, which in some cases also extends to user directory customizations.

Numerous SPIs in NetBeans are already defined to use the SFS for registration: everything from the contents of the main menu bar, to user XML catalogs, to project Properties dialogs. Switching all these to another system would be massively incompatible; making all such SPIs read both the SFS and another source would be tricky; and using the new registration system only for newly written APIs would be confusing and not accomplish the objective of simplifying the platform.

The current proposal is therefore "lipstick on a pig":
it hides the complexities of NetBeans service registration
behind a simple facade to make the life of the average module developer easier.
If you want to create your own SPI, debug a strange problem in service loading,
inject objects such as menu items into the platform dynamically using Issue 26338,
use advanced branding to modify standard registrations, or just understand what is going on,
you will need to understand both the old complex internals as well as the new annotations.
For the same reasons, it is not likely that we would be able to treat an annotation
as the sole supported way of using an SPI;
it would still be necessary to document and support the underlying SFS syntax.

Build-time OpenIDE-Module-Layer injection

The current solution to generation of layer fragments from annotations
is for processors to create META-INF/generated-layer.xml
and for the runtime (mainly the module system) to look for this resource
in addition to any traditional declared layer.

An alternative strategy would be to modify the <jarwithmoduleattributes> Ant task
to combine generated layer fragments with an explicit source layer, if there is one,
creating a new OpenIDE-Module-Layer declaration on demand.
The advantage would be to avoid any changes to the module system.

The reason for the decision to not follow that approach is to avoid making additional assumptions about the build environment.
Modifying that Ant task would work well enough for modules built from scratch
using the official build harness.
But one of the main features of JSR 269 is that it can work
with no modification to any build process: you just run javac (in JDK 6+).
Consider:

Modules built using Maven should get annotations processed automatically (assuming JDK 6+ is used to run Maven).

Developers doing incremental development in other IDEs which bypass Ant should still be able to use the new annotations, provided the IDE is 269-compliant.

Processors in separate JARs

The current design assumes that annotation processors just live directly in module JARs.
In the case of layer-generating processors, the processor would conventionally
be kept in the same module as the annotation it processes,
would would generally be in the same module as defines the associated SPI.

An alternative would be to require that processors reside in separate JAR files
(to be loaded using -factorypath when running javac).
While this could make module JARs a little bit smaller and avoid certain dependencies,
it would make it much more cumbersome to define a processor.
It would also complicate the support for Compile-on-Save, Maven, etc.

Note that processors need to compile against JSR 269.
Since this is part of JDK 6 but not 5,
for now modules with processors need to either add a regular module dependency
on org.netbeans.libs.javacapi, or add this JAR directly in cp.extra.
(The build harness loads a 269-compliant compiler in either case.)
When JDK 6 becomes the minimum supported platform for NetBeans,
it will be possible to simplify the build harness
and no special dependency will be needed in modules providing a processor.

There are some complications.
A special Self ID refers to a populated panel with children.
Sometimes there is a panel category that itself has no content.
This still needs a display label and position, yet is not associated with Java code.

Actions

@ActionRegistration is available.

Problems with actions

There is no clear place to register the display label of a menu.
As discussed above, this problem predates the use of annotations,
since multiple modules contribute to a menu yet they must agree on one label.

There is no obvious place to register menu separators.
Letting the annotation specify separators "before" and "after" the menu item
is the current solution, but this may not work well
with modules wishing to cooperate on separator locations.
One possibility, inspired by Eclipse's system,
is for infrastructure to automatically insert separators between blocks of menu items,
where modules can somehow agree on what the blocks are;
most simply, declare that separators will be placed automatically
at positions 0, 1000, 2000, etc.

Keyboard shortcuts are often specific to a profile (Keymaps folder).
Which modules should register these and how?

OpenProjectFolderAction above knew about its target data type,
but often the situation is reversed: one action is referenced from many modules.
For example, a data loader wishing to construct its context menu
will list an EditAction, CopyAction, and so on.
In the case of a SystemAction in org.openide.actions.*,
these can just be referred to by well-known class name.
How could a more general action be listed from a data loader?
Clearly the action class itself cannot be annotated
for a data loader about which it knows nothing.
The data loader registration annotation needs to list its actions, but using what syntax?