Using Reinforcement Learning To Learn To Play Tic-Tac-ToeAbout a year ago I set myself the goal of writing an algorithm that could learn to play tic-tac-toe. I didn't want to tell the algorithm what the rules of the game are, nor did I want it to try and use some kind of calculation to look ahead at possible ...

The best opening move in a game of tic-tac-toeAs part of a machine learning project, I had to understand tic-tac-toe better, and so I have written an algorithm which a) finds all the possible unique games and b) gathers statistical information about those games. Based on Wikipedia's tic-tac-toe ...

I recently participated in a course where we learned about Eclipse e4 development. It was all very interesting, especially the reliance on declarative services from OSGi. I decided to have a think about how to build a skeletal application which makes server calls, to show how I would design the application architecture and break the application into bundles (plugins/components). I also focused on MVC.

The app is a simple realtime market place for selling animals. Users can view the details of animals which are for sale and then purchase any animal they like. They can also add their own animals to the market place for others to purchase. Here's what it looks like (the red text isn't part of the app!):

The first step was to think about a few requirements, use cases and design points:

Refresh List - reload the list of available animals by making a server call to the central database. The user can view the list of freshly loaded animals and by clicking on an animal they can view it's details, like price, age and name.

Purchase Animal - While viewing an animals details, the user may attempt to purchase the animal. It is only an attempt, because in a concurrent system, another user may have already purchased the animal. The client only shows the last loaded state - there is no push notification mechanism from server to client to inform it that there has been a change in the central model. As such, the clients model is simply a snapshot of the central model, taken at the point when the client loads the model. Purchasing the animal which costs above a configurable threshold requires additional security checks via the supply of an ID number. The client should not be forced to call the server in order to determine if the ID number is required, because the system should respond quickly to the users inputs, because the market place is a realtime environment.

Sell Animal - The user can provide details of any animal which they want to sell. They input the details and once they click the OK button, that animal is added to the market place and is available for purchase by any user.

So, to implement such a system, using Eclipse e4, I developed the following bundles. You can click the image to enlarge it.

So, there are plenty of points to make about that diagram... Each block in the diagram is a bundle (plugin). The block sometimes shows a stereotype, then in larger letters the bundle name (abbreviated), and finally a list of components or classes which are the main reason the bundle exists.

App - The app is a simple e4 application which only contains the central Application.e4xmi model, the product file, the icons and the CSS. It contains NO implementation details like classes! The purpose of this bundle is simply to bring it all together in an application. As such it has no direct dependencies on other bundles, it simply requires that at runtime the classes it uses in the XMI model be present within the OSGi environment.

UI - This bundle contains all the UI classes that the application requires, and as such builds the view used in MVC. It contains the Parts which e4 uses to build the screen, as well as dialogs, and the handlers, which e4 commands use. The UI has two dependencies. The first is a dependency on the controller which is used any time something slightly complex needs to be done. When for example, the user wants to purchase an animal, the controller is passed the selected animal and it checks if an ID (passport) number is needed, and then makes the server call to do the purchase centrally. The second dependency is to the model and domain objects. The view has read only access to the model, because I want all changes to the model to be done using the controller. See below for details on the model, but it provides a read only interface using OSGi declarative services, and it is this read only interface which is injected by the e4 EAP into our view (part). The UI consists of two Parts. The left hand Part is the list, the right hand Part shows the details of the selected animal. When an item is selected in the list on the left, the org.eclipse.e4.ui.workbench.modeling.ESelectionService is used to inform any listeners in the application that something was selected. The detail Part binds itself to the selection service during construction and is hence informed whenever the user selects an animal from the list, so that the detail part can update itself and show the animal's details.

Domain Model & Model - The model consists of two bundles. The domain model is where classes which are used both serverside and client side are created. Things like Dog and Cat and their superclass Animal are contained in the domain model. The server and client both use these objects. Some might say the client shouldn't use these objects (it should use transfer objects instead) but as long as they are not the objects used in the persistence layer (i.e. the JPA entities) and as long as the client and server are tightly coupled by design (i.e. it really makes sense), then I have no problem with shared objects on the client and server. Now, since these objects are used on both sides of the wire, classes related to event notification do not belong along side them. So, all such classes are placed in the Domain Model bundle. But since the client is event driven, we need a mechanism to fire events and listen to them. The org.eclipse.e4.core.services.events.IEventBroker is an ideal mechanism, but I didn't want to build a dependecy from my model to Eclipse. So the solution was to use java.beans.PropertyChangeSupport to fire change events and handle event listeners. To use the IEventBroker in the UI (view) I gave the controller the responsibility to push events from the property change support to the event broker. I guess this is a step that some might consider skipping, because the model isn't the domain model, there is no real reason it cannot depend directly on Eclipse.

The next important note about the model is that it has two different interfaces. The first, the IModel is a read only interface of the model, intended for use by the UI. The view should never directly change the model, but always do it's changes using the controller. The controller in turn needs an interface which it can use to modify the model, say after validation or a server call. This is the Model interface, which gives more access to the model implementation. The actual model implementation is in a class called ModelImpl. The model is made available using OSGi declarative services, and so can be injected into the controller and view.

Note that the domain model classes do not fire events. That means that any changes to the model need to be done using the Model interface, so that the model implementation can fire the required events. Say you want to add an animal to the model. You don't get the list from the model, and add to the list, because no event would be fire. What you do is call a method on the model which adds the animal to the list and also fires the relevant event. True property change support means that any leaf or branch in a model tree which is changed, leads to a fired event. I find such designs too fine grained, and the number of events which are fired becomes overwhelming, leading to poorly performing UIs which are constantly redrawing themselves. While there are ways to limit the redrawing (see some of my other blog articles), I feel that you don't want to start programming all that stuff if you can avoid it from the start by using a coarsely grained event firing design.

Core - The core bundle provides the controller. The controller has the responsibility to check any changes which the UI wants to make to the model and dealing with the changes. After any verification that is required, and after performing any business logic which is required to be on the client (as little as possible!), and after calling the server if it needs to, the controller uses the Model interface of the model to push the changes into the model. The model then fires an appropriate event (using property change support) which is passed via the controller straight to the event broker, which the UI listens to and uses to update itself if it needs to.

Up to this point, all dependencies between the view, model and controller were injected by the e4 EAP. But I had one problem... I tried making the controller an OSGi declarative service and giving it a dependency on the model. That worked, but I had no way to access the event broker to bind the model to it. The solution was suggested by a colleague, and it was to create an e4 Addon. The addon is configured in the e4 XMI file in the app bundle and is simply a class which is instantiated once when the application starts, and has access to the IEclipseContext. So it uses the ContextInjectionFactory to make the controller, which allows the controller to have the model and event broker injected into it. The addon then puts the controller into the context, so that the UI can have it injected too.

The picture above has an "IView" in the core bundle. That component doesn't exist anywhere in the source code, but is there to show how the controller talks to the view. Normally a controller only does stuff with the view by firing events using the model. But in some cases, it may need to show an error to the user, or get more information from them. Those kind of things need the UI, JFace, SWT, etc. Instead of throwing exceptions which the UI can catch and handle and then recall the controller once the problem is resolved, you might want the controller to directly call the view. A good case is when we need to get the ID number from the user, but only in certain cases, which the controller must evaluate. But the controller should be loosely coupled from the view - in MVC it should always be possible to exchange the view with no impact on the system. In order to decouple the view from the controller, we use interfaces, and "IView" is in the diagram to show such interfaces. In the source code (available at the end of this article), you can see such an example in the IdProvider interface, which has lots of Javadocs talking about its use.

How does the controller call the server? Well the controller only knows about an interface called the IAnimalService. Where or how that interface is implemented is not interesting to the controller. But the controller does need a way to locate that service, and it defines an interface called the IServiceLocator. In true OSGi declarative services style, the implementation is looked up at runtime, when the controller wants to get hold of the service. This means that the service implementation is entirely pluggable and is a simple deployment issue. If you want a stub service which returns dummy results for say testing errors, then you deploy the bundle containung that service implementation. If you want to call the server, then deploy the real bundle. In our case, the service locator implementation uses another OSGi declarative service which is part of a library I have written (and which is the subject of a further blog article to come later). That library lets you send serialized Java objects or even EMF models over the wire to a server using normal HTTP(S). This library is very similar to other products such as Caucho's Hessian or other such libraries supplied with Spring Remoting.

Server - The server is a simple web application which uses the same library which the service locator uses to call the server. The library deserializes the request and sends it to the relevant service which is implemented on the server. The library handles sending the response back to the server too. Since the server uses the same domain model as the client (i.e. the domain classes, not the actual object instances!), the webapp has the domain model JAR and service interface JAR in it's classpath.

I'm quite happy with the solution that I created. While it doesn't make use of all the new e4 features, it does show how to use it's dependency injection for an MVC implementation and how to use declarative services to create the non-UI objects. Central things like the event broker and selection service are used too.

One point where you could change the design slightly is that it is arguable whether the IModel and Model interfaces belongs in the model bundle, or whether they belong in the core bundle. It is the controller which requires them, so why couldn't the core bundle define this interface, and using OSGi declarative services, let the deployer ensure it is available when the controller needs it? The model bundle contains the model implementation, and both these interfaces. The service locator interface on the other hand is not in the bundle which provides the service locator implementation. That's because the core bundle author is stating that they simply need the IServiceLocator interface, and don't care who provides it's implementation, so long as it's available at runtime.