Invoice subsystem

Driven by the entitlement and subscription modules, along with the catalog configuration, the invoice subsystem is core to Kill Bill. This manual will walk you through a couple of use-cases, to help you understand how it works under the covers and how to make sense of the data on disk.

Overview

Before we dig into examples, let’s review some of the characteristics of the invoicing algorithm. The first thing to recall is that there are different ways to invoice customers and this is specified inside the catalog configuration. For the sake of simplicity we will discard usage-billing altogether here and focus on recurring subscriptions (along with fixed-term charges).

Invoicing for recurring subscriptions also comes in two different fashions, or so called billing modes. Such a mode is defined at the catalog level, and there are 2 options:

IN_ADVANCE: In such a configuration, each recurring subscription is charged at the begining of the period (in advance), and as a consequence, this can lead to pro-ration credits upon certain cancellation or change of Plan (special policies can be configured to make the system generate or not such pro-ration credits).

IN_ARREAR: In such a configuration, each recurring subscription is charged at the end of the period (in arrear), and as a consequence there is never any pro-ration credit generated because the system always know what to bill for (things can’t change back in time).

In the rest of this documentation, we will assume an IN_ADVANCE billing mode because this is the most common and most complex to deal with.

Let’s revisit some of the terminology:

chargedThroughDate or CTD: We call chargedThroughDate the last day of the recurring period associated with a given subscription that has been charged for; there is one such date for each subscription in the system.

targetDate: We call targetDate the date up to which we want the invoiving system to bill for. Let’s assume a monthly subscription and a current date of 2012-05-01 (which corresponds to our PHASE event in our example below), and let’s look at the impact of the targetDate:

2012-06-01 ⇐ targetDate < 2012-07-01 : System would bill for period [2012-05-01, 2012-06-01) and [2012-06-01, 2012-07-01) and CTD = 2012-07-01

Kill Bill always invoice at the account level, meaning each time an invoice is being generated, this is for a given Kill Bill Account (matching a given customer) and so the resulting invoice will include potentially different items matching different subscriptions associated with this particular account. In the most normal case, invoices are generated by reacting to specific events and we 'll see examples below of the system reacting to specific bus events or (future) notifications.

Initial subscription creation

Let’s assume a new user subscribes to the shotgun-monthly plan on 2012-04-01.

Associated with the subscription are a couple of subscription events, one CREATE marking the beginning of the subscription at the effective date 2012-04-01 00:01:14 and one PHASE event marking the date at which the EVERGREEN phase starts:

The entitlement subsystem has also a record of the start of the entitlement, in the blocking_states table (on older Kill Bill versions, this was not present so you could still see some data were this is missing and this is fine, the system knows how to handle this case):

Upon subscription creation, a bus event is triggered and caught by the invoicing subsystem, which invoices the account with a target date of 2012-04-01. To do so, it computes the billing events from these subscription events and blocking states (they are effectively markers between billable periods). In our case, these billing events are:

The target date being 2012-04-01, only the first one matters. Based on the catalog configuration, the following invoice and invoice item are generated (an invoice has always 1 or more invoice items associated with it):

What happens is that the payment subsystem calls the createPurchaseWithPaymentControl API and specifies INVOICE_PAYMENT_CONTROL_PLUGIN as the control plugin to use (you can add your own via the system property org.killbill.payment.invoice.plugin). This plugin is responsible to compute the payment amount (typically the invoice balance) and to insert a row in the invoice_payments table (success is first set to false, to implement a two-phase commit strategy). The payment is then delegated to the payment system (a payment and/or a transaction are recorded if necessary). Upon success, the entry is updated with the metadata from the payment (the plugin could have decided to pay less than the requested amount for example) and the success flag is set to true. In case of failure, success would remain set to false and the next payment retry date would be computed, based on the system configuration.

While the link between invoice and payments is encapsulated in the invoice_payments table, there is one level of indirection with the payments table through the payment_attempts table, to manage aborted payments and retries:

Aborted payments: In a situation where the invoice was already paid (or there is a $0 balance), the invoice control plugin would abort the payment. In such situations, we would end up with a row in the payment_attempts table with an ABORTED state and no row in the payments and payment_transactions tables.

Payment Retries: In a situation where we see a payment failure (e.g. insufficient funds), a payment will be associated with multiple transactions (all sharing the same transaction external key and typically in a PAYMENT_FAILURE status). Each of these transactions will be associated with an attempt in a RETRIED state.

Note also that the payment_attempts entry is linked to the invoice via the plugin property IPCD_INVOICE_ID (which points to the invoice id).

In our scenario, no payment was actually processed, since the invoice amount is zero (trial). Hence the ABORTED state. See below for an example of an actual payment and what would happen in case of payment failures.

Reference time and fixed offset timezone

Each account in Kill Bill has a reference time associated with it. It is the reference_time value associated with the accounts row (specified at account creation time, default value is the created_date). Additionally, it is associated with a fixed offset timezone, a special timezone used by the system for dates manipulation which ignores changes like DST (you cannot change it either). This timezone depends on the account timezone (UTC if none specified) and if DST was in effect at the reference time (this lets us handle gracefully DST gaps throughout the year with respect to invoicing, handling subscription changes, etc.).

For example, if the account timezone is America/Los_Angeles and the reference time is 2015-03-07T10:00:01.000Z, DST was not in effect and the fixed offset timezone is -08:00. If the reference time however is 2015-03-08T10:00:01.000Z, DST was in effect and the fixed offset timezone is -07:00.

While most subsystems (entitlement, subscription, etc.) work at the time level (i.e. you could have several upgrades during the day), invoicing works at the day boundary level (Kill Bill doesn’t invoice for granularities smaller than a day). Both of these parameters help us convert LocalDate (i.e. a specify day in a year, like 2012-04-01) to a DateTime (i.e. a specific point in time, in a specific timezone), and vice versa. The former conversion is required for instance when computing the next notification time (based on the invoice item end date for instance). The latter can happen during invoice generation, when transforming a BillingEvent effective DateTime to a LocalDate for the invoice item (service period).

A nice side effect of using this fixed reference time is that most system-driven operations will always happen at the same time during the day for a given account. It also helps spreading the load on the system since the distribution of these reference times should be uniform.

Phase transition

Let’s fast forward the time to 2012-05-02.

The notification for the phase event is processed by the subscription subsystem. There is nothing to be done in that case (in other scenarios, add-ons may need to be cancelled or a future phase event may need to be computed): it simply sends a message on the bus letting the system know about the phase transition.

The invoicing subsystem picks it up and re-compute the billing events:

Nothing has changed but since the target date is now 2012-05-01, both events need to be taken into account. The invoice subsystem recomputes all invoice items since the beginning of time, and come up with a FIXED item (trial period) and a RECURRING item (for the service period 2012-05-01 to 2012-06-01). Because the FIXED item is already present in the database, only the second one is persisted on disk, on a new invoice:

In this case, a PURCHASE (i.e. auto-capture) payment was performed. The invoice_payments entry is linked to the payments entry via payment_id and to the transactions table via payment_cookie_id (which is the transaction external key).

If the payment didn’t go through the first time (e.g. insufficient funds on the credit card), and the system was configured to retry the payments 8 days after, the data would look like this on a successful retry:

The ITEM_ADJ of $-10 points to the recurring item (see linked_item_id). Because the balance of the invoice was $0, a credit item (CBA_ADJ) of $10 is also added and will be available when the next invoice is being generated.

Refund with invoice item adjustment

A variation of the invoice item adjustment is refund with invoice item adjustment, i.e. refund the customer instead of generating a credit.

With a START_OF_SUBSCRIPTION change alignment, both timelines align on the start of the subscription (2012-04-01). On 2012-05-02, the target phase is hence the DISCOUNT one. If we had chosen a CHANGE_OF_PLAN alignment instead, the blowdart-monthly timeline would have been aligned on the date of the change (2012-05-02) and the target phase would have been the TRIAL one.

Two new subscription events are recorded, one for the change and one for the future phase change:

The REPAIR_ADJ item on the new invoice points to the RECURRING item on the second invoice that is being adjusted (repaired). The amount repaired is only $239.95 because of the previous item adjustment.

Because the new rate is only $9.95, the account has extra credit of $239.95 - $9.63 (pro-rated) = $230.32 (new CBA_ADJ item).

Note that the previous invoices have not been updated in that case.

A new payment attempt is triggered, but aborted since the invoice balance is $0:

Invoicing internals

The invoice subsystem computes new invoices in several stages.

First, it loads all existing invoices on disk for the account being processed, and creates a tree per subscription (SubscriptionItemTree) containing all of these invoice items. Each node in the tree represents all invoice items for a given service period. The service period of a parent node overlaps all service periods of its children.

For example, let’s assume a subscription was billed in advance from 2012-05-01 to 2012-06-01, and a change plan occured on 2012-05-07. There would be 3 items on disk, one recurring for the first plan (2012-05-01 to 2012-06-01), a second recurring for the second plan (2012-05-07 to 2012-06-01) and a repair adjustment (from 2012-05-07 to 2012-06-01) associated with the initial item.

A is the root node, it doesn’t contain any item and the service period is the largest period containing all items in the tree. Node B represents the initial recurring period (the node has one ADD — or A — invoice item). B has one child node — C — which has two items, the ADD item representing the new recurring item, and the CANCEL item, pointing to the ADD item in the node B, representing the repair.

The next step is to build the tree: resulting invoice items for each service period are computed and item adjustments processed (if any). In this case, 2 items would be generated (one from 2012-05-01 to 2012-05-07 and one from 2012-05-07 to 2012-06-01). The original tree is then replaced (flatten phase) by a shallow one containing these items (it’s really a list at this point):

The items already invoiced are also reversed and labeled as C (CANCEL) before the next step.

Billing events are then processed and a list of proposed invoice items is generated. Each item will either be matched to an (existing) item in the flattened list, trigger a repair (in case of new change plan for instance) or kept as-is.

For example, if a proposed RECURRING item from 2012-05-01 to 2012-05-07 is merged, the flattened list becomes:

A new node D, child of c, is inserted representing that period. After the RECURRING from 2012-05-08 to 2012-06-01 is merged, the list would be unchanged (no match found and no repair to create) and the RECURRING item kept on the side.

Finally, one last time, the tree is built and the computed invoice items become the resulting items, i.e. the items for the resulting invoice. In our case, two items are generated: the RECURRING kept as-is (outside the tree), and a repair from 2012-05-08 to 2012-06-01.

Keeping invoices in DRAFT mode

In some situations, we would like Kill Bill to generate the invoices, but leave them in a DRAFT mode.

Examples:

Case where we need to append invoice items after generation

Case where we want payment system to ignore the invoice until it has been COMMITTED

This behavior can be configured at the account level through the AUTO_INVOICING_DRAFT tag: when an Account is tagged with AUTO_INVOICING_DRAFT, any automatic invoice generated by the system (such as when generating invoices for recurring and usage subscriptions) will end up in DRAFT mode. Generating invoices through API calls (e.g. adding external charges) will not be affected by this tag.

Implementation expectation

Any DRAFT invoice containing RECURRING or USAGE items will be taken into consideration by the system when computing subsequent invoices, regardless of the AUTO_INVOICING_DRAFT tag.

As an example, let’s assume a monthly recurring subscription billed on the 1st of the month:

AUTO_INVOICING_DRAFT is added

January 1st : DRAFT invoice generated with a RECURRING item from January 1st to February 1st

AUTO_INVOICING_DRAFT is removed

February 1st : COMMITTED invoice generated with a RECURRING item from February 1st to March 1st — although the previous invoice has not been committed yet

In other words, the feature assumes that at some point such DRAFT invoices will be committed by the user. Not doing so will lead to users not being billed for those specific periods.

Invoice Date

The invoice date is the one from when the DRAFT invoice was generated — the date at which this invoice gets committed is not reflected. In particular, this may have some impact if the payment associated with such invoice fails, and dunning kicks-in: the timeSinceEarliestUnpaidInvoiceEqualsOrExceeds overdue configuration option will be based on the original invoice creation date (DRAFT).

Re-using invoices in DRAFT mode

The account level AUTO_INVOICING_REUSE_DRAFT control tag will make the invoicing system reuse an existing DRAFT invoice instead of creating a new invoice when generating invoices for recurring and usage subscriptions, or when generating parent summary invoices:

If there is no existing DRAFT invoice, the system would default to normal behavior and create a new invoice

If there is one existing DRAFT invoice, the system would use this DRAFT invoice instead of creating a new one

If there is more than one, the system will use the earliest DRAFT invoice

Similar to the AUTO_INVOICING_DRAFT feature, it is the responsibility of the user to to commit such DRAFT invoices.

Parked Accounts

Because invoicing drives payments, any issue in the system or misconfiguration leading to wrong invoices could have a direct impact on your customers. In order to avoid bad invoicing (and potentially invalid customer payment) we have introduced a mechanism to PARK the accounts. Once an account has been parked, no more invoicing will happen until its state has been fixed (or until safety bounds have been removed). A account can become parked in the following scenarii:

First, the system now has a safety bound mechanism on the upper limit of invoice items generated for a given subscription on a given day. This safety bound is controlled by a system property org.killbill.invoice.maxDailyNumberOfItemsSafetyBound whose default value is set to 15 but that can also be disabled by setting it to -1. The safety bound will kick-in if a customer tries to buy too many things during a single day or in situations where the customer has not been invoiced for a while and the system is catching up (could happen if the account was tagged with AUTO_INVOICING_OFF and the tag gets removed).

Second, the invoicing system will automatically park an account when an illegal state is reached during invoicing. This could be because the data associated with the account looks corrupted (which could happen because of previous bugs that did corrupt the data, manual data manipulation, or existing bug in the system).

The way to get out of that state is to fix the issue that lead to that state in the first place and trigger a new invoice generation. For cases where there are many parked acounts, we have added an administrative api to go through all parked accounts and issue an invoice generation for each of those, only leaving in the parked state, accounts whose state has not been fixed yet. If you think the issue that lead into that state is not a problem with the account state (data) but a bug in the system, please report to the mailing list.

Summary

The Kill Bill invoicing system generates invoices on a per account level based on some system trigger:

Bus event: Those are usually subscription events like when a subscription starts, when there is a phase transition (trial → evergreen), upon plan change or cancellation (there are also other bus events in the system that could trigger an invoice generation, but not discussed for sake of simplicity).

Future notfications: The recurring piece of the invoiving is based on the invoice system inserting future notfications on the right date (so as to be called back at the right time and generate the next invoice).

Note also that invoice generation can be triggered through API but this is not the default use case.

The computation of the next invoice (and matching invoice items) is based upon the following:

A billing mode (e.g IN_ADVANCE)

A set of billing events (mostly aggregated across subscritpion_events and blocking_states tables)

A targetDate specifying upon which point to bill (and which billing events to consider)

The invoicing algorithm recomputes the full view at each invocation and compares the existing view on disk (what was already billed) with the new set of proposed items, and based on the difference generates the next invoice (or nothing if there is no change since the previous invocation). If a new invoice was generated, it also inserts a new future notification and updates the chargedThroughDate for all subscriptions.

In terms of payments, the source of truth resides in the payments (and associated payments_transactions) tables. However, the payment view of the invoicing subsystem relies on the invoice_payments table and is used to compute things like invoice and account balance: The invoice_payments table is updated using a 2-phase commit algorithm where a row is first inserted with a status set to false and then upon payment completion is updated to true.