Notification Plugin: A plugin registered to receive bus events. When a bus event is being delivered to the plugin, the plugin calls back Kill Bill through API calls to retrieve additional state for this event or to change the state in the system

Plugin that registered as an implementor of a specific plugin API: A payment plugin for instance would have registered itself as an implementor of the PaymentPluginApi and the payment system would then invoke that plugin for issuing payment operations. The plugin would have the ability to call back Kill Bill using APIs to retrieve more state or create/change state in the system.

A Kill Bill plugin will either implement one (or several) plugin API(s) and/or also listen to Kill Bill bus events to be notified of changes and take appropriate actions. It often makes sense to specialize plugins and have them implement only one of the plugin APIs, but this is a design choice, and there is nothing preventing a plugin from implementing multiple of those APIs. Although we don’t recommend it, a plugin could implement the PaymentPluginApi and the PaymentControlPluginApi, and that would qualify it as a 'payment and payment control plugin'.

Kill Bill Plugins can be used in a variety of ways:

Extend Kill Bill subsytem: Payment plugins are a good example of such plugins; they extend the core payment functionality by connecting to third party systems

Provide additional business logic: Payment control plugins and invoice plugins allow to intercept the requests so as to modify it, abort it, or change state on the fly.

Listen to system events: The analytics plugin is a good example of such plugin

Kill Bill Plugins have full power (and therefore need to be designed and tested carefully):

They have access to the database so as to maintain their own state. However they should not interact with the Kill Bill core tables directly but rely on APIs to retrieve and change state,

They have access to system properties

They have the ability to export their own HTTP endpoints under a /plugins/<pluginName> namespace

They receive all events in the system

They are isolated from the rest of the code and can use their own libraries (versions) without risk of conflict

Where to Start?

A good starting point is to assess what the plugin should do and then based on that, read the various plugin docs in the developer guide section that describe the different types of plugins offered in Kill Bill:

The next stage is to identify existing (similar) plugins which could be used as a starting point to write the code. At this point, this becomes a normal software development cycle, including writing unit tests for that plugin (independent of Kill Bill).

Development

It is recommened to clone some of our existing plugins to get started. For java, the hello world plugin is a good starting point. For ruby, we also provide a hello world plugin. Depending on the type of plugin to be built, there may be a better example to look at (e.g. look at an existing payment plugin if you are building a new gateway integration).

We provide two plugin frameworks that can be used to implement some of the work that plugins need to do - although your plugin does not have to rely on these frameworks, it is often a good idea to leverage those to avoid boilerplate code.

Also, for internal reference, you might want to take a look at KillbillActivatorBase, which provides all the abstractions that plugins require (access to java APIs, database connections, system properties, logging, …​).

Java Plugins

Prerequisites

Java versions 6+ are supported.

Build

java plugins rely on the maven-bundle-plugin to produce the correct OSGI plugin (with the correct MANIFEST.MF). It is recommended to start from a plugin that already works, as the build configuration can be quite tricky. A good example is to look at the pom.xml from the hello world plugin

The build will produce a jar under the target directory which should contain all the classes and configuration files directly accessible for that plugin. OSGI also provides a way to import classes that are exported by other plugins and this is configured in the Import-Package section of the maven-bundle-plugin. The hello world plugin provides some default based on the classes exported by the system bundle in Kill Bill. Kill Bill offers a system property, org.killbill.osgi.system.bundle.export.packages.extra, to specify additional packages to be exported by the system bundle and that could in turn be imported by the plugin. Desiging which packages are served by the plugin jar and which one are imported from another bundle (and mostly the system bundle inside Kill Bill) is one of the challenges of the OSGI mechanism.

Once the pom is correctly written, building the plugin is straightforward:

Deployment

After the plugin was built, one can deploy the plugin using the kpm command. The steps provided below assume Kill Bill is running inside a docker container and specify the steps that occur outside and inside the container:

Note that the last step that restarts the container — and therefore Kill Bill core — is required to make the newly installed plugin visible. KPM only allows to install the plugin on the filesystem and until Kill Bill core gets restarted this plugin is unknown to the system. However, once the plugin has been installed once, iterating through development becomes easier because all is needed is to copy the jar at the right location and restart the plugin — restarting Kill Bill core is not even necessary anymore — as illustrated below:

Note that when using OSS released plugins, the steps are simpler because the plugin can be installed directly using our plugin management apis and don’t require mucking around with copying jars and restarting Kill Bill core.

Deployment

The steps are very similar to the one provided in the java section above, except for the fact that in ruby we are not deploying a jar but the package (tar.gz). Assuming a modified version of our stripe plugin was built:

Examples of Ruby Plugins

We provide a hello world ruby plugin that can be used as starting point. Make sure to correctly update the *.gemspec and the pom.xml to correctly reflect the gem name, and maven coordinates of your plugin (if you decide to publish your plugin to Nexus).

We also provide a collection of Ruby Snippets, which shows how to call Kill Bill APIs from Ruby plugins.

Recipes

Using NotificationQueue from plugins

There are times when a plugin needs to queue some events for later/future processing. Such situations could be to handle certain bus events at a later time, or enqueue specific operations that have failed so they can be retried, …​

In those situations, the plugin implementor has the choice to use its own mechanism, use open source solutions,…​ and this all depends on the objectives (e.g persistent queue?) and the familiarity with certain frameworks.

One possibility is to use the NotificationQueue abstraction that is already provided by Kill Bill framework.

One needs to keep in mind that the use of such notification is rather internal to Kill Bill core but in some situations it is acceptable to use it from plugins with the following understanding:

This is should not be used for high throughput/low latency kind of situations, as the mechanism fundamentally stores events into the database and allows to be dispatched to a specific handler

Improper use of such queue can have bad consequences for the rest of the system

It forces the plugin to know low level details (e.g accountRecordId)

Only a small number of queues should be created (1 or max 2 for a plugin) as they create some load on the system (each queue will have its own pool of thread for dispatching event notifications).

The goal of this documentation is to provide some guidelines on how to make it work.

Queue Creation

Services can create a new queue and later use that queue to send (event) notifications and process such notifications.

The creation of the queue will require the following parameters:

svcName : A string that identifies the service (a good convention is the plugin name)

queueName: The name of the queue (a good convention is to use a string that describes its function)

handler : The handler that will be called back each time a notification is dispatched. The handler should be idempotent as the system ensures at least one delivery

The NotificationEvent interface is a marker interface; the class implementation must be serializable using json. An example can be found here.

userToken is a UUID of your choice that is passed when publishing events.

searchKey1must be the accountRecordId attached to the account or null

searchKey2must be the tenantRecordId attached to the tenant or null (if operations is cross tenant, but very unlikely)

Note that accountRecordId and tenantRecordId are usually not visible from plugins (instead the plugin will see the accountId and the tenantId which are UUID. However there is 1-1 mapping between those two ID (one being internal and the other being user visible) and info can be retrieved using the following APIs:

Queue Configuration

Each queue that is created at runtime by the system (whether a plugin or Kill Bill core), needs to have its own set of tables in the database. By convention, we usually name such tables in the following way:

{tablename}_notifications

{tablename}_notifications_history

Where tablename is the name of the plugin or something related that will be unique and easily understood to be related to the given plugin. This value comes from the configuration associated with that queue.

Create the NotificationQueueConfig by replacing the instanceName with the desired value (e.g plugin name)

Create the DefaultNotificationQueueService specifying the config for that queue

Create the handler

Create the queue

Lifecycle

Before this can happen the queue needs to be properly started and also stopped when the plugin stops.

Starting the queue will start the pool of thread that is attached to the queue. The system will print a trace showing that such threads were started. Such threads should be visible by running a jstack command and looking for the following name: config.getTableName() + "-th".

Of course such lifecycle operations should also match the lifecycle of the plugin, the start and stop function defined in the activator. Note that failure to start the queue will have the effect to see such events in the IN_PROCESSING state but handler will never be called.

For more information on configuring the queue, please refer to this document.

For PAYMENT events, the payment id can be found in event.object_id and the tenant in event.tenant_id

Deployment

Overview

Plugin Layout

In its simplest form, deploying a plugin means placing the plugin binary at the right place on the filesystem. Kill Bill will scan the filesystem on startup and will start all the plugins that were detected.
Kill Bill will use the value of the system property org.killbill.osgi.bundle.install.dir to determine the root of plugin directory structure.
By default, this value is set to /var/tmp/bundles, as indicated by the Kill Bill OSGIConfig file.

jruby.jar : the Runtime JRuby jar that is loaded into killbill for each ruby plugin

A set of pure OSGI bundles (unrelated to Kill Bill plugins) and required for things like OSGI logging, OSGI console, …​

Under java and ruby, we will find one entry per plugin per version.
For instance, if we had installed two versions for the ruby stripe plugin, we would see the following (SET_DEFAULT is a symbolic link that point to the default active version):

ruby
|_killbill-stripe
|_ 3.0.2
|_ 3.0.1
|_ SET_DEFAULT

The file sha1.yml is a used by the KPM tool to keep track of artifacts that were already downloaded to avoid dwonloading binaries already present on the filesystem. KPM also offers the --force-download to override that behavior.

The file plugin_identifiers.json is used to keep a mapping between the pluginKey (the user visible plugin identifer), and the pluginName (runtime identifier used by Kill Bill when scanning the filesystem). The next section provides more details about those.

Plugin Coordinates, Plugin Key, Plugin Name, …​

Today, both our ruby and java plugins are released through maven and are therefore identified through their maven coordinates. We might support other schemes in the future but today this is the only way we offer to download and install publicly released plugins. Plugin Coordinates are too cumbersome to manipulate though and are unsuitable for non-published plugins (typical use case for a plugin being developed), so we introduced some identifers.

As mentioned earlier, Kill Bill will scan the filesystem (org.killbill.osgi.bundle.install.dir) on start-up to detect and then start all the plugins. The name on the filesystem (e.g. in our previous example killbill-stripe) constitutes what we call the pluginName.

When installing using KPM, the pluginName is dependent on how the plugin was packaged and also differs between ruby and java. For well known publicly available Kill Bill plugins, we adopted a (sane) convention, but we have no way to enforce that convention for third party plugins. Also, note that we could change the name of killbill-stripe to foo on the filesystem (mv killbill-stripe foo) and then suddenly Kill Bill would see that plugin as being the foo plugin. Therefore, the pluginName is not a reliable way to identify the plugin, and is used solely by Kill Bill as an runtime identifier.

The pluginKey is the identifier for the plugin and is used for all the user visible operations, whether through the KPM command line tool or whether using the Plugin Management APIs.
There is a distinction to be made between publicly released Kill Bill plugins and third party plugins:

(Publicly Released) Kill Bill Plugins: All the plugins developed by the Kill Bill core team are maintained in a repository (we provide today a simple file-based repository, but this may change in the future as we start accepting certified third-party plugins).
Each entry in that repository is identified by a key, and that key is the pluginKey.

Third party plugins: For third party plugins, the key is specified at the time the plugin gets installed. The key must be of the form <prefix>::<something> to make sure there is no name collision with Kill Bill plugin keys.

Deploying by Hand

Java Plugins

For java plugins, deploying by hand consists of building the self contained OSGI jar, and copying that jar at the right location. For example, the adyen plugin with a version with version 0.3.2 would show up as the following:

java
|_adyen-plugin
|_ 0.3.2
|_ adyen-plugin-0.3.2.jar

Ruby Plugins

For ruby plugins, deploying by hand consists in building the package (tar.gz) and untaring that package at the right place: For example, the stripe plugin with a version 3.0.2 would show up as the following:

In order to make it easy to deploy those plugins we created a special rake task that will copy and untar plugin entries at the right place:

# Deploy the plugin (and clobber a previous version if needed) in /var/tmp/bundles.# Alternatively, you can manualy deploy the .tar.gz or .zip artifact from the pkg/ directorybundleexecrakekillbill:deploy[true]

Note that if you don’t need any custom configuration, make sure to delete the default YAML configuration file /var/tmp/bundles/plugins/ruby/killbill-*/*/*.yml. In development mode, i.e. when you are running tests outside of Kill Bill (see rake test:spec and rake test:remote:spec), the database configuration is specified in that YAML file (payment plugins rely on a couple of database tables, principally to keep the credit card tokens and gateway-specific details for transactions, such as reference codes). By default, the plugin will use SQLite. If you uncomment the part of the YAML file below the comment "In Kill Bill", this will tell the plugin to use the JNDI connection exposed by Kill Bill instead. This is the default in case the file isn’t present (or if the database section is missing).

Also, in the case of ruby plugin (and as mentionned before), the correct version of the jruby.jar must exist (and be named that way) under the platform directory entry. The correct version must match the Kill Bill version (or more precisely the version of the platform used for the specific version of killbill).

Deployment Through KPM

The standard way to deploy plugins is to rely on KPM.
The KPM README explains how to install KPM and also provides some guidance on how to deploy publicly released Kill Bill plugins.

Plugin Configuration

System Properties

Kill Bill plugins can access Kill Bill properties through the use of a special interface OSGIConfigProperties. System properties passed to the JVM and properties from the killbill.properties configuration file are then accessible to the plugins and can be used to tweak the behavior of the plugin as needed.

Configuration File

Property files can be used to configure global settings for a plugin. Those property files need to be part of the archive (the OSGI mechanism will make sure these are only visible to the particular plugin):

For java plugins, the property file needs to be on the classpath (resource directory)

For ruby plugins, the property file is often located at the root of the archive

There is no restriction on the format of the propery file, but typically ruby plugins will use yml files and java plugins will rely on key-value properties, json or xml files.

Per-tenant Configuration

The two previous mechanisms work well for global settings, but are inadequate to configure the plugins on a per-tenant fashion (e.g for a payment plugin interacting with a payment gateway, different credentials might be needed for different tenants). In those situations, Kill Bill provides APIs to upload/retrieve/delete per-tenant plugin configurations: