Monday, July 30, 2007

Now you can download BRMS standalone version, this distribution comes with a built-in Tomcat 5.5.20 web server and the insurance example as demo repository, so it runs out the box. If you want to test BRMS and don't have enough time to deploy, just follow to downloads page http://labs.jboss.com/drools/downloads

Wednesday, July 25, 2007

JBoss Drools 4.0 has just been released :) We are really proud of what we have done here. We believe that we now have the best and most powerful declarative rule language, bar none; commercial or open source. The Rule Flow is excellent and I really enjoyed updating the Conway's Game of Life to Rule Flow; sub Rule Flows and milestone support will be coming in a point release soon. The BRMS has long been requested and we put a lot of effort into the ajax based design. The Eclipse improvements for the debug points and guided editor should help reduce learning curves, opening us to new audiences. Of course performance is now much better, especially for complex rules and the new Sequential Mode should be very popular with decision services.

Here is a more detailed enhancements list:Language Expressiveness Enhancements* New Conditional Elements: from(hibernate ready), collect, accumulate and forall* New Field Constraint operators: not matches, not contains, in, not in, memberOf, not memberOf* New Implicit Self Reference field: this* Full support to Conditional Elements nesting, for First Order Logic completeness.* Support to multi-restrictions and constraint connectives && and ||* Parser improvements to remove previous language limitations, like character escaping and keyword conflicts* Support to pluggable dialects and built-in support to Java and MVEL* Complete rewrite of DSL engine, allowing for full l10n* Fact attributes auto-vivification for return value restrictions and inline-eval constraints* Support to nested accessors, property navigation and simplified collection, arrays and maps syntax* Improved support to XML rules* Experimental Clips parser support

Monday, July 23, 2007

I've had a few people mention their unhappiness with the lack of backwards compatible in 4.0 to 3.0.

Yes I do apologise about the changes, but I do feel that if we want to progress towards the best engine, it can't be helped - I do offer to assist the community in developing backward compatible wrappers, or upgrade scripts, no one ever volunteers.

2.0 really had to be scrapped, it just wasn't a real expert system language - a compatibility layer could have been written and we put up details on how to get started on this, but no one was interested.

3.0 was a total rewrite for a "proper" expert system, but time constraints and limited understanding meant some compromises where made. With 4.0 we had the opportunity to get this all right and I now believe we have the best declarative rule language, bar none, a language that we can live with and extend for some time.

There are a number of changes that will cause 3.0 upgrade problems, I detailed these in email 3 times and also a blog entry. We also do regular milestone releases, so people can see what we are doing, and document the backward comparability issues in the release notes. I only got one feedback saying "go for it". Again I blogged and emailed about the start of an upgrade script, zero interest of volunteers from the community.

We want to make the best rule engine/language if we break something, it's because something isn't right; to live with that will be, longer term, strategically worse than breaking it now.

To give you an idea of this decision making consider in 3.0 we auto-boxed all primitives (as we couldn't handle primitives directly) and didn't auto-unbox - this was an ugly situation - so people had to handle all their own unboxing. Now in 4.0 we handle primitives directly, allowing people to write cleaner code; however all the manual unboxing they did will now be bust; which in many code bases is considerable.

So we have a choice we can keep a really ugly limitation of 3.0, or we can break it in 4.0 (hence the major number change) and get it right which results in much cleaner rules. For me the choice was clear, break 3.0. There is also the argument that all breakages should be "deprecated" and switches added to enable legacy execution - which is what some commercial firms do - this is an expensive undertaking and we simply don't have the resources we are a small team and rely on community involvement. If we continue to build up layers of cruft for legacy support, we simply won't be able to innovate at the pace we do. I prefer instead to create upgrade paths which is absolutely feasible, and zero impact on the main codebase; however as I mentioned previously people only want to complain about the changes, and not do anything to assist in making it work, which is a shame :(

When 4.0 final comes out and suddenly all these users that don't read my blog, don't read the mailing lists, don't participate in the community try and upgrade and realises it breaks there systems they are going to give us stick for it; probably publicly in various forums. I hope those reading those forum postings can point them here and to the upgrade script we want to build and encourage them to help solve this.

We will work on those scripts, and they will get better over time, but if we want those fixed quickly we need your help.

Sunday, July 22, 2007

"We are using Drools VERY succesfuly around 3 years (We started with version 2.x). And it has being so usefull that now we have a very big system running on more than one customer in telecom market....Currently we do have a server running around 300 rules, where we assert millions of facts at once. The objective is to guide and rate telecom usage events. We are able to apply those 300 rules over 20 Milion facts and get results (Guided and Rated) around 1 hour(Note: First we have to read many binary files, perform a charset conversion on data, load it in our object model, assert objects in working memory, apply around 200 rules to enrich the data, assert again in a new working memory, and rate the events accessing external RDBMS databases - caching results of course)."

"We tested several deployment and architecture variants (esp. for batch processing), with <100 Rules and <10 facts for each data-row within our batches, but a few million rows. Worked out fine."

"We deployed a Drools based solution to a client in the pharmaceutical distribution world (Fortune 100 company :-). We used Drools to power the decisions an interactive voice ordering system. A small number of rules initially but growing constantly based on user metrics. System has been running for a couple of years now."

Saturday, July 21, 2007

I've created a dedicated Drools job board, http://jobs.athico.com, which will be linked from this site; a small widget on the right shows the 5 most recent job postings. The board is currently free and unmoderated, it will remain unmoderated while recruiters do not abuse it. At the moment I'm still waiting for the first job post, so hurry :)

To create a more cohesive feel I've set-up a cname for the job board, the blog and the feeds:http://blog.athico.comhttp://jobs.athico.comhttp://feeds.athico.com/DroolsRSShttp://feeds.athico.com/DroolsAtom

athico.com was just some short easy to remember domain name I had hanging around. The previous feed addresses will continue, although I encourage subscribers to move across, the blog address http://markproctor.blogspot.com and article links will re-direct - so it should be a pain free move.

Friday, July 20, 2007

As promised I finally got round to updating the popular Conways Game of Life example so it can be executed with both agenda-groups and ruleflows for execution control. So now people have a good example to study when trying to learn the two concepts. (click to enlarge images).

With agenda-groups we specify the stack execution order for each group, in the java code.With ruleflow we instead specify the process id of the ruleflow:Here you can see the ruleflow for the above process id, notice how "birth" and "kill" are executed in parallel:This code is in trunk, and will be part of the next release for drools-examples.

From working on this example what has become obvious is we need support for sub ruleflows, so that we can have a parent ruleflow choreographing all the ruleflows in the application, which is currently being done from java code.

Wednesday, July 18, 2007

Having received community feedback 4.0 will start the push to reclaiming the Drools name. We are now updating the jboss.org site to have Drools instead of JBossRules everywhere, the new logo is up and we have also put the logo up at the blog. We will be putting out a single release, Drools, so you won't find jbossrules*.zip for 4.0. The product side, for now, will still be called "JBoss Rules", simply because you can't change that side of things over night, sales people have to be retrained, sales material has to be updated, partners have to be communicated too - but in essence JBoss Rules is just JBoss Drools + support. So to help speed up this process, I would please encourage everyone now to use "JBoss Drools" in any articles, blogs, talks etc.

A good blog entry from Dr. Gernot Starke, I love the opening paragraph, which can't be said enough :)"Welcome to real life. Don't ever believe that rules can and will be written by business-people only."

Another useful titbit is to use an Agenda Filter with a Stateless Session. You use the AgendaFilter to isolate the rule(s) you wish to test. The Agenda Filter can be setup just once on the session and then collections of data can be executed against the session. You can then change the Agenda Filter when you wish to test a different rule.

StatelessSession session = ruleBase.newStatelessSesssion();

session.setAgendaFilter( new RuleNameMatches("<regexp to your rule name here>") );

Monday, July 16, 2007

Now that we are close to a release, and many people are looking for info on the new stuff, I've put together an essential readings blog entry, the two marked with a * I would consider particularly interesting. Enoy :)

Friday, July 13, 2007

There is a new BRMS example available. It is written up in the manual as a tutorial, but the information is repeated here. We are having some issues with our build server - so the docs available in the snapshots are probably not as up to date as the source is.

-------------Fernando runs a dodgy fly by night insurance operation in Sao Paulo, Brasil. These are his rules - would you buy insurance from him ? (I wouldn't).

This example takes you through the key steps in using the BRMS with an example respository, and using rules in a very very simple application (which you can use as a basis for your applications).

Thursday, July 12, 2007

The SkillsMatter session last night was a full house, people even came back after the half way break, the fools ;) So thank you everyone for making the effort to come and listen; I hope you all enjoyed it.

SkillsMatter should be putting up the presentation video soonish, however I don't think it will be worth while watching, I couldn't get the video project to display my presentations properly; all the text and lines where pixelated and colour wasn't showing through :(

It's really hard to get across the basic theory of an expert system and then demo enough of the system that the audience feels they have a basic understanding of the capabilities of the system - in just two hours! I hope I got a reasonable balance. Anyway SkillsMatter have asked me if I'd like to come back and do some specialist sessions, to which I said yes - so expect me to come back to spend 2 hours talking about specific aspects of the Drools, for those already with the basics and wanting to understand more; please let us know what you'd like to see more of.

Wednesday, July 11, 2007

Just a reminder that my skills matter is tomorrow, here is the original posting:

I'm going to be at SkillsMatter on Wednesday July the 11th (tomorrow) doing a seminar on JBoss Rules 4.0. The location is near Farringdon tube station and the time is from 18.30 to 20.30, and afterwards I'll be around for drinks at the pub. So please do try and come along.

Monday, July 09, 2007

Bruno Duarte mentioned GoogleFight as a humorous way of looking at the popularity of keywords for Drools vs "JBoss Rules", for fun I also plugged in "Blaze Advisor" and JRules and pasted the screenshots below. Of course the results are pretty meaningless, especially as Drools is also a generic word; but it was still fun watching the fights :)

After 18 months of the JBoss Rules name for the product, Drools name for the project, my gut feeling is the split is contrite and only causes confusion - something that Marc Fleury also expressed at the time.

The Drools brand was much stronger than expected, most sites use the two words together "JBoss Rules/Drools", "JBoss Rules aka Drools", "JBoss Rules (Drools)", looking at Jobserve shows both words are used in all adverts - this to me shows a confusion in the market, with little advantage from the dual branding. Some places still use just Drools, this is also pushed by competing vendors that leach from us and want to hide any JBoss association - although this is undesirable it is still happening and causing confusion; there isn't much we can do to improve that situation, so might as well ride the wave - otherwise we risk something worse, lose of brand association of Drools with JBoss. On that note several vendors upstream what we do, none have given a single line of code or documentation back to be project; which is a pretty sad state of affairs, they obviously feel it helps their bottom line to include the software, yet won't lift a finger to help improve it - shame on you.

JBoss Rules as a name is poor for building a community, as it's too formal - people wouldn't like to say "JBoss Hibernate" or "JBoss Seam" all day, it's fine as a title or in the opening sentence, but then you want to switch to the informal name, otherwise it gets tiresome. Drools makes a good name space naming, and a prefix/postfix naming convention for classes. It's a single word and single syllable, making it easy to aggregate into other product names - "JBoss Rules BRMS" is a bit of a mouthful "Drools BRMS" is much easier.

There is of course those that don't like the Drools name, for its childish innuendos, and I did consider re-branding to something like DRules. I feel however that a further name change would be weak, and just cause even further confusion, especially with the strength of the Drools brand. So that would just have to be something we learn to live with :)

For these reasons, and many more I didn't mention, I am considering using JBoss Drools name for the 4.0 release. Feedback welcome :) even if it's just a +1 with your choice, to help me better guage public opinion.

Sunday, July 08, 2007

When talking people through the Drools 4.0 set of features and how they work, I've found that there is simply too much for people to grok, without previous Expert System experience. So I thought I'd turn to Mind Maps to see if I can help make all our advanced concepts more grokkable. I found a great GPL tool called FreeMind. I'm using version 9-beta9, which seems to be stable enough to use. I've put down an initial brain dump to get started, the idea is map all our concepts with notes and links through to the main documentation or other related sites for more info.The FreeMind file for the above map is in subversion, the documentation/mindmap folder, FreeMind also does flash exports, which I've committed, so you can play with it live now by clicking here, and give me your feedback.

Friday, July 06, 2007

With Rete you have a stateful session where objects can be asserted and modified over time, rules can also be added and removed. Now what happens if we assume a stateless session, where after the initial data set no more data can be asserted or modified (no rule re-evaluations) and rules cannot be added or removed? This means we can start to make assumptions to minimise what work the engine has to do.

Algorithm

Order the Rules by salience and position in the ruleset (just sets a sequence attribute on the rule terminal node).

Create an array, one element for each possible rule activation; element position indicates firing order.

Turn off all node memories, except the right-input Object memory.

Disconnect the LeftInputAdapterNode propagation, and have the Object plus the Node referenced in a Command object, which is added to a list on the WorkingMemory for later execution.

Assert all objects, when all assertions are finished and thus right-input node memories are populated check the Command list and execute each in turn.

All resulting Activations should be placed in the array, based upon the determined sequence number of the Rule. Record the first and last populated elements, to reduce the iteration range.

Iterate the array of Activations, executing populated element in turn.

If we have a maximum number of allowed rule executions, we can exit our network evaluations early to fire all the rules in the array.

The LeftInputAdapterNode no longer creates a Tuple, adding the Object, and then propagate the Tuple – instead a Command Object is created and added to a list in the Working Memory. This Command Object holds a reference to the LeftInputAdapterNode and the propagated Object. This stops any left-input propagations at assertion time, so that we know that a right-input propagation will never need to attempt a join with the left-inputs (removing the need for left-input memory). All nodes have their memory turned off, including the left-input Tuple memory but excluding the right-input Object memory – i.e. The only node that remembers an assertion propagation is the right-input Object memory. Once all the assertions are finished, and all right-input memories populated, we can then iterate the list of LeftInputAdatperNode Command objects calling each in turn; they will propagate down the network attempting to join with the right-input objects; not being remembered in the left input, as we know there will be no further object assertions and thus propagations into the right-input memory.

There is no longer an Agenda, with a priority queue to schedule the Tuples, instead there is simply an array for the number of rules. The sequence number of the RuleTerminalNode indicates the element with the array to place the Activation. Once all Command Objects have finished we can iterate our array checking each element in turn and firing the Activations if they exist. To improve performance in the array we remember record the first and last populated cells.

The network is constructed where each RuleTerminalNode is given a sequence number, based on a salience number and its order of being added to the network.

Data StructuresTypically the right-input node memories are HashMaps, for fast Object retraction, as we know there will be no Object retractions, we can use a list when the values of the Object are not indexed. For larger numbers of Objects indexed HashMaps provide a performance increase; if we know an Object type has a low number of instances then indexing is probably not of an advantage and an Object list can be used.

Everything to the above paragraph is in JBoss Rules 4.0, the following sections are idea for the future, and thus still a bit vague at the moment.

If there is a huge number of rules a indexed paged direct address table can be used in place of the array that holds the resulting Activations. Here we create pages of arrays, we create further index points to pages at regular points in the range. The page can indicate if any of its elements are populated or not, allow us to skip iterations of those elements.

Collapsing Nodes, Code Generation and Node OrderingStateful Rete can have nodes added and removed, for this reason to maximum node sharing we tend to have an AlphaNode per literal constraint. However as we know we will not have any more rules added we can collapse shared nodes into a single evaluation. If we have a chain of 3 AlphaNodes A, B and C. where A and B are shared and C isn't. We can collapse A and B into a single node and generate code to do the multiple evaluations natively. Node ordering can be changed to maximise sharing and thus the impact of Collapsing them. There may also be some level of beta node collapsing we can do, but I have'nt given that side much thought yet.

Exploiting Multi-ThreadingIn normal Rete exploiting multiple threads is hard, as you never know if a Tuple is going to enter the left-input or an Object is going to enter the right-input – i.e. reads and writes can happy simultaneously, the effect of adding locks to control this is expensive and past research has shown the costs are more than using lock free single thread approach. However with our sequential approach we know that the writes will always happen at different time to the reads. All writes happen during the first stage where objects are propagated to the right input nodes. All reads happen during Tuple propagation from the LeftInputAdapterNode. We can exploit this to execute parallel threads for each stage; using concurrent collection structures. We only have handle the synchronisation of the two stages – i.e. knowing when the all the Object propagations are finished and knowing when all the Tuple propagations are finished. Executing the actual rules in parallel is a little harder as we cannot easily know the user intention of the sequence. An attribute can be used to execute groups of rules that can be executed in parallel; those rules and groups must be specified in chronological order. i.e. If group A is to fire before group B, then all the rules for group A must be specified first. Or the execution order for the groups can be specified seperately, allowing rules to be more free formly tagged with the attributes.