crumb

Crumb is an annotation processor that exposes a simple and flexible API to breadcrumb
metadata across compilation boundaries. Working with dependencies manually is usually fine, but there's often
cases where developers will want to automatically gather and act on information from those dependencies (code
generation, gathering metrics, etc). Tools like ServiceLoader can solve some cases like this, but lack flexibility and
can be slow at runtime.

This is where Crumb comes in. Crumb's API is an annotation-based, consumer/producer system where extensions
can opt in to consuming or producing metadata. Extensions run at compile-time to produce or consume this
metadata, while Crumb's processor manages this metadata for them (serializing, storing, retrieving, orchestrating the
data to appropriate consumers, etc). This allows developers to propagate arbitrary data across compilation boundaries.

API

Annotations

@CrumbProducer - This annotation can be used on custom annotations to signal to the
processor that elements annotated with the custom annotation are used to produce metadata.

@CrumbConsumer - This annotation can be used on custom annotations to signal to the
processor that elements annotated with the custom annotation are used to consume metadata.

@CrumbQualifier - This annotation can be used on custom annotations to indicate that elements
annotated with the custom annotation are relevant for Crumb and used by extensions.

@CrumbConsumable - A convenience annotation that can be used to indicate that this type should be
available to the Crumb processor and any of its extensions (since processors have to declare which
annotations they support).

Extensions API

There are two extension interfaces that follow a Producer/Consumer symmetry. The API (and compiler
implementation) is in Kotlin, but seamlessly interoperable with Java. The API is SPI-based, so implementations can be
wired up with something like AutoService.

Both interfaces extend from a CrumbExtension base interface, that just has a method key(). This
method has a default implementation in Kotlin that just returns the fully qualified class name of
the extension. This is used to key the extension name when storing and retrieving metadata.

The API usually gives a CrumbContext instance when calling into extensions, which just contains
useful information like references to the ProcessingEnvironment or RoundEnvironment.

CrumbProducerExtension - This interface is used to declare a producer extension. These extensions
are called into when a type is trying to produce metadata to write to the classpath. The API is:

supportedProducerAnnotations() - Returns a set of supported annotations. Has a default
implementation in Kotlin (empty), and is used to indicate to the compiler which annotations
should be included in processing (since annotation processors have to declared which annotations
they need).

isProducerApplicable(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror> -
Returns a boolean indicating whether or not this producer is applicable to a given type/annotations combination.
The annotations are any @CrumbQualifier-annotated annotations found on type. Extensions may use
whatever signaling they see fit though.

produce(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror> -
This is the call to produce metadata, and just returns a Map<String, String> (typealias'd in
Kotlin to ProducerMetadata). Consumers can put whatever they want in this map (so be
responsible!). The type and annotations parameters are the same as from isProducerApplicable().

CrumbConsumerExtension - This interface is used to declare a consumer extension. These extensions
are called into when a type is trying to consume metadata to from the classpath. The API is:

supportedConsumerAnnotations() - Returns a set of supported annotations. Has a default
implementation in Kotlin (empty), and is used to indicate to the compiler which annotations
should be included in processing (since annotation processors have to declared which annotations
they need).

isConsumerApplicable(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror> -
Returns a boolean indicating whether or not this consumer is applicable to a given type/annotations combination.
The annotations are any @CrumbQualifier-annotated annotations found on type. Extensions may use
whatever signaling they see fit though.

consume(context: CrumbContext, type: TypeElement, annotations: Collection<AnnotationMirror>, metadata: Set<ConsumerMetadata>) -
This is the call to consume metadata, and is given a Set<Map<String, String>> (typealias'd in
Kotlin to ConsumerMetadata). This is a set of all ProducerMetadata maps discovered on the
classpath returned for this extension's declared key(). The type and annotations parameters
are the same as from isConsumerApplicable().

CrumbManager

Crumb's core functionality can be leveraged independently from the compiler artifact via the
crumb-core artifact. This can be useful for integration within existing tooling, and contains
a CrumbManager and CrumbLog API. The crumb-compiler artifact is an advanced frontend
over this utility.

CrumbManager has a simple load and store API, and CrumbLog is a logging mechanism to help
with debugging issues.

Kotlin Issues

Note that this is only when consuming data in a plain Kotlin project. Android projects work fine.

Example: Plugin Loader

To demonstrate the functionality of Crumb we will have a hypothetical plugin
system that automatically gathers and instantiates implementations of the Translations interface from
downstream dependencies. Conceptually this is similar to a
ServiceLoader, but at
compile-time and with annotations.

To prevent a traditional approach of manually loading the implementations, Crumb makes it possible to
automatically discover and utilize the Translations classes on the classpath.

Producing metadata

The plugin implementation then needs to be registered into the plugin manager upstream. A Crumb
extension can convey this information to consumers of the library by writing its
location to Crumb and retrieving it on the other side. For this example, a custom @Plugin
annotation is used to mark these translations implementations.

@CrumbProducer
public @interface Plugin {}

Note that it's annotated with @CrumbProducer so that the CrumbProcessor knows that this
@Plugin annotation is used to produce metadata. Now this annotation can be applied to the
implementation class:

Crumb will take the returned metadata and make it available to any extension that
also declared the key returned by key().

context is a holder class with access to the current ProcessingEnvironment and
RoundEnvironment

type is the @CrumbProducer-annotated type (EnglishTranslations)

annotations are the @CrumbQualifier-annotated annotations found on that type. For
simplicity, all holders are required to have a static obtain() method.

Consuming metadata

For the consumer side, our example will have a top-level TranslationsPluginManager class that just delegates to
discovered downstream translations. With a ConsumerExtension, downstream services can be consumed
and codegen'd directly with JavaPoet. For simplicity, this manager will follow an auto-value style
pattern of having an abstract class with the generated implementation as a subclass.

Crumb can be wired in here. The symmetric counterpart to @CrumbProducer is @CrumbConsumer, so
this example uses a similar @PluginPoint annotation here for consuming. This time it's annotated
with @CrumbConsumer to indicate that it's for consumption.

This closes the loop from the producers to the consumer. pluginClasses contains a set of all
downstream plugin type implementations and could leverage JavaPoet to generate a backing
implementation that looks like this:

Note that both extension examples are called PluginsCompiler. Each interface is fully
interoperable with the other, so it's possible to make one extension that implements both interfaces for
code sharing.

The complete implemented version of this example can be found under the :sample:plugins-compiler
directory.

There's also an example experiments-compiler demonstrating how to trace enum-denoted experiments
names to consumers.

License

Copyright (C) 2018 Uber Technologies
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.