Namespaces

A Clojure implementation of a simple forward chaining rule engine. An engine is created by defining rules in one or more modules and invoking engine.core/engine on keywords defining the modules. Each working memory element in the engine is a Clojure map. The name "Arete" is a pun on the greek word and "a-RETE" (i.e. not RETE), since the engine is based more on the TREAT algorithm than RETE.

The result of invoking :run [<wme>...] on an engine is to insert the specified working memory elements (wmes), run rules until no more will fire and then return a map of wme types to sequences of wmes.

This engine was originally created to help with translating between different container orchestration formats (Kubernetes and Docker Compose). Both formats were changing rapidly and they don't have particularly similar structures. Rules made it easy to express global constraints and to avoid a lot of complicated and fragile navigation code. Though there are multiple Java-based rule engines, they are quite heavyweight, and are mostly oriented more toward expressing business rules than just providing an additional programming paradigm. The only Clojure engine we found (Clara Rules) is a great tool but is aimed at different use cases. It shares with this engine, however, the advantage that rules can be expressed directly in a Clojure program without any new, separate language that needs to be parsed.

Since this engine was used for translation (and not, say, cluster management) there is not much support built in for having an engine instance run forever while taking new inputs and processing them. It would actually be relatively easy to do and will probably happen at some point if there's interest.

The Arete engine is available from clojars as [arete "0.6.1"] or simply download the repo, install leiningen if necessary, and run lein uberjar. The main class in the uberjar is the rule viewer for debugging. The engine itself has no command line.

(<engine> :run[-map] [<wme>...]) - Run to completion after inserting wmes. After running, clear the engine state so that a subsequent call will encounter a fresh engine. Returns a map of wme types to collections of wme instances.

(<engine> :run-list [<wme>...]) - Run to completion after inserting wmes. After running, clear the engine state so that a subsequent call will encounter a fresh engine. Returns a list of wmes.

(<engine> :cycle [<wme>...]) - Run to completion after inserting wmes. After running, leave the engine alone so subsequent calls add to the state rather than starting fresh.

The basic conflict resolution strategy of the engine is simple priority. However, there is a declarative means of specifying preferences. A rule module can contain a "deforder" expression specifying how conflicts should be resolved:

(deforder (:with :x) (:without :y) :oldest)

The expression above says that any instantiation containing a wme of type :x should be preferred over one that does not contain an ":x" and if neither one contains an ":x", pick the one without a ":y" over one that does contain it. Finally, if all (or no) instantiations contain an ":x" and all (or no) instantiations contain a ":y", pick the instantiation that was created first. The set of currently available checks is:

Sometimes it's useful to write rules that operate on abstract categories of wmes that are otherwise of different types. Maybe you want to write a rule that deals with all "shapes" instead of specific "circles", "squares", etc. This is supported in the engine by the use of "defancestor" expressions. The following:

says that any rule that matches a ":controller" should also match a ":deployment", ":daemonset", ":statefulset", or ":cronjob". The ancestor relationship is transitive so any ancestor of ":controller" would also be an ancestor of its descendents.

This shows us at step 0 with one rule instantiation and two wmes (the _start wme used to trigger rules without left hand sides and our factorial argument). The "*" after each number indicates that the wme or instantiation was newly created. Let's type in the number of the argument to get a better look:

(0)==> 2
WME - (2):factarg
value: 6
(0)==>

Not too much to see here since it's a very simple wme. A '?' will tell us all our options:

(0)==> ?
Usage:
'<':
go to beginning
'>':
go to end
'?':
display this help
'.':
exit the viewer
'<cr>':
if at top level, move forward one firing; otherwise return to top level
'<number>[,<number>]*':
display insts or wmes with <number>s as ids
'ar':
display all rule firings for the run
'b':
back up one firing
'e <exp>':
evaluate expression referencing an individual wme as: :<id> and all wmes
as :0
'g <step id>':
go to step number: <step id>
'h':
display command history
'pi <str>':
display partial rule instantiations for rules with name containing <str>
'r':
display rule firings leading to this point
'ref <wme id>':
display any wmes that reference the specified wme via a UUID link
'rs <str>':
display rule with name containing <str>
'sc <wme id>':
find the firing that created the specified wme
'sd <wme id>':
find the firing that deleted the specified wme
'ss <str>':
find the next firing containing a wme whose string representation includes
<str>
'st <type fragment>':
find the next firing containing a wme whose type name includes the <type
fragment>
'sr <rule fragment>':
find the next firing for a rule whose name includes the <rule fragment>
'si <type fragment>':
find the next firing whose instantiation references a wme with type
containing the <type fragment>
'save <filename>':
save the history to a file (when running as ":record true")
'w':
display all wmes for current firing
'w <type fragment>':
display all wmes for current firing with types containing <type fragment>
'ws <str>':
display all wmes for current firing whose string representations contain
<str>
(0)==>

Timing of individual rules and more global properties of the engine can be obtained by setting the environment variable "DEBUG_COMPILE" to true and configuring the :enable-perf-mon flag to true. This will cause performance stats to be collected which can be viewed by invoking :timing on the engine after a run. With no argument or an argument of false, the timing data will get cleared after being requested. An argument of true will leave the data intact so it can be combined with data from other runs. The result of calling :timing is a CSV file written to stdout that can be imported into a spreadsheet. Something like:

Name, Total Time, Number of Invocations, Max Time, Min Time, Mean Time

Currently, the engine internal data is mixed in with the data for user rules. At some point, the perf interface will be made more user-friendly; however it is already quite useful in its current state.

The performance monitoring has very little overhead, particularly if the engine does not have it enabled and DEBUG_COMPILE was not set. However, if you want the absolute maximum speed for a deployed system, set the environment variable NO_PERF_COMPILE to true before building your application.

doesn't need to compare every instance of :some-type with every instance of :some-other-type. Instead, the wmes for :some-other-type are stored in a hash map keyed by the value of :some-other-field. When it's time to process an instance of :some-type, it is directly combined with instances of :some-other-type that result from a hash lookup of its :some-field value. In general, this is always worth doing as it completely avoids the normal cross product comparisons.

The arete engine also implements a less common optimization that is not always useful, but can be extremely so. Consider the following rule:

This inserts 9999 striped balls, 9999 solid balls, and 5 gurks. The values for striped and solid balls only overlap in one entry (0).

When run on my laptop, this test takes nearly two minutes to run because each insertion of a gurk forces a full cross product evaluation of the two ball matches. This is something of a worst case scenario for arete since it does not save intermediate join results between executions (A RETE-based engine would have a similar problem if a new wme was added in a match higher in the LHS and would do much more work if "balls" rather than "gurks" were being added and removed). If we make one minor change, however:

it runs in less than 500 milliseconds. The optimization applied here inserts entries being joined via an inequality into Java TreeMaps so that the engine can iterate over the headMap of the map containing entries that match the inequality. The process has a fair bit of overhead, so it isn't automatically applied. To trigger the optimization, replace '>' with '>>', '<' with '<<', '>=' with '>>=' and '<=' with '<<='. To see the worst case for the optimization, consider:

When run with '>=', this takes ~1.3 seconds. With '>>=', however, it takes: ~2.3 seconds. So, there's a tradeoff. If you're operating on large numbers of objects and you see slow behavior around an inequality, try replacing the operator with its TreeMap equivalent and you may see a significant improvement.

The engine is loosely based on the TREAT algorithm, though the handling of negation is different (probably worse...) However, it does correctly handle negated conjunctions. For rules without negation, the processing is quite efficient with very little allocation (only the instantiations and maps to hold wmes). Negation requires maintaining a much more elaborate tracking structure. Like TREAT, no intermediate state is saved for beta tests (other than hashes of values so that we can avoid cross-product performance).

No attempt is made at making this a purely functional implementation; Java maps and other data structures are used throughout for maximum efficiency. If immutable sessions or truth maintenance are important for your application, check out "Clara Rules" instead.

To explain how the engine works, we'll go through a briew overview and then build up from the simplest case. Most forward chaining rule engines use some variation of the RETE algorithm. The RETE algorithm was originally developed based on the insight that the firing of a forward chaining rule typically leaves most of the working data unchanged. This suggests that it's worthwhile to precompute matches and hold on to them between firings since most of the work will not need to be redone. The RETE algorithm takes this perspective to the limit by precomputing and caching everything it can including the results of comparisons between fields of distinct objects (i.e. joins). As it turns out, though, there is a significant amount of bookkeeping overhead associated with maintaining precomputed joins. TREAT (and this engine) discard join results and recompute them as necessary. A RETE engine feeds working memory elements into the top of a discrimination network and "rule instantiations" come out the bottom. TREAT places working memory elements into maps and then works from the bottom up to find matches. Compiling the rule:

and two sets of functions, one for managing the alpha maps (adding and removing wmes) and one to manage rule matching and instantiation. The functions that manage the maps invoke the root function from the second set for each associated rule.

The map functions are pseudocode because they're constructed as closures programmatically and many of the variables they reference are defined in the functions that create them. The two map functions look different because the cube function uses an extra level of hashing to allow efficient comparison of ball radii with cube sides.

When a new working memory element is added (e.g. a ball), each alpha (map) function associated with the wme type is called and, if it results in adding a new element, the current map for the type is replaced with a map containing only the new element. Then, the main function of the rule is called and it performs joint matches which will only include the one new wme for the type (since all other cross matches will have already been done when the other values were added). Afterward, the map is set back to the original map plus the new element.

Negated object matches are far more complicated and will be discussed in detail below.

Completely general rule conditions require the ability to express arbitrary boolean logic. This engine provides that support in the form of a "negated conjunction" or "nand" match. It's well known that either "nand" or "nor" by itself is sufficient to express any boolean logic, so we add the ability to express nested nands on left hand sides. Consider a loan application rule preventing too many lines of credit from being allowed against a given mortgage. Along with other rules preventing too much leverage, we have a rule that says no more than two lines of credit per mortage:

The branch going up to the right represents the nested "nand" with two line of credit matches. The "*" is the nand node. It takes action any time execution begins from the bottom of the graph and begins in "pass" state. If a new mortgage wme is added, the wme is stored in the mortgage alpha node for the rule and execution is started from the bottom. When execution reaches the nand node (in "pass" mode) it passes execution up the left side and does the following for any successful upstream matches:

Bind outer-vars to all the object match variables in any outer network. In this case, that's just one variable: ?mortgage.

Run the entire inner nand subnetwork and collect any resulting negative instantiations.

If there are any resulting instantiations, store them in a new nand record for the nand node keyed by the wmes; otherwise, call the downward function with the wmes.

If an outer wme is removed, the nand node will not get triggered as all the instantiations, nand-records, and other related data will get removed directly without invoking the network.

If a wme is added to the inner network, we have to look for matches with each wme combination that made it to the nand node. So, we iterate through the nand records and, for each one:

Set the nand node to "sub" mode

Bind outer-vars to the wmes.

Run the inner network and if there are any resulting instantiations:

If there is already a nand record for the wmes, add the instantiation to its instantiation list and stop; otherwise, create a new nand-record, add the instantiation, and remove all positive instantiations downstream whose wmes include the wmes present at the nand node as a prefix.

If a wme is removed from the inner network, we remove all instantiations and set the nand node to "rem" mode and, when control reaches it from the bottom, we retrieve all nand records that match and no longer have any instantiations. The wmes contained in these nand records are passed to the downstream function.