Using Reinforcement Learning To Learn To Play Tic-Tac-ToeAbout a year ago I set myself the goal of writing an algorithm that could learn to play tic-tac-toe. I didn't want to tell the algorithm what the rules of the game are, nor did I want it to try and use some kind of calculation to look ahead at possible ...

The best opening move in a game of tic-tac-toeAs part of a machine learning project, I had to understand tic-tac-toe better, and so I have written an algorithm which a) finds all the possible unique games and b) gathers statistical information about those games. Based on Wikipedia's tic-tac-toe ...

I'm bored of reinventing the wheel. Everytime I need a state machine to ensure my states traverse only valid transitions, I find myself either not bothering, because I trust my coding (and write all the necessary unit tests of course), or writing very similar code over again.

So I started wondering if there was a configurable state machine out there somewhere, and in no time at all Google gave me a link to SCXML from Apache. Apache SCXML is an implementation of a configurable state machine based on the SCXML working draft from W3C.

I started by taking a look at what it does and how it works, always keeping in mind my requirements based on previous projects. The main question was how I could use a state machine in a persistent entity so that when an attempt is made to change the state, the state machine validated the attempt, ensuring only valid transitions are carried out. That meant two things:

The state machine had to be able to have its current state set to any state. If I load an object
with state out of the database, I need to be able to set that state in the state machine so that
it checks any attempts to change state, based on this starting state.

The state machine had to fit into a JPA entity class so that I could persist and load the state.

Apache SCXML doesn't come with great documentation but if you look around, it has some "use cases" which are examples of how you can use it. It comes with the class org.apache.commons.scxml.env.AbstractStateMachine which uses reflection to trigger transitions. I went for a slightly different approach and created the uk.co.maxant.demo.scxml.util.AbstractStateMachine, which wraps an instance of the SCXMLExecutor (the "engine", or state machine itself). By wrapping the state machine, I am able to provide two extra benefits:

I can construct the state machine using the starting state out of a persisted entity, rather than
being forced to use the state charts initial state. The implementation details of this construction
are a little complicated so my abstract wrapper can hide the details from the caller who is more
interested in writing some business code, rather than boiler plate code.

I can enforce only valid state transitions, in that before performing a transition, I can use the
state machine to check if its a valid transition from the current state. Apache SCXMLs default
handling means that if you try to set the state to an illegal state, the state machine simply does
nothing. I prefer to be a little harsher and throw an IllegalStateException!

The implementation of the AbstractStateMachine contains trigger methods for changing state via transition events. Each event makes a simple call to the AbstractStateMachine.trigger(String) method which first of all checks that the current state allows the transition, and second of all delegates the transition to the wrapped state machine instance. If the requested state transition is illegal, then an IllegalStateException with details of the problem is thrown. I too, could have also used reflection like the Apache example, but prefer to write more readable code for this Blog.

Apache SCXML is built up by creating an SCXML instance which is based upon the configuration which is an XML representation of the state chart and can be generated from UML. As such, this configuration has an initial state, as defined in your state chart. You then create an instance of the engine (state machine, SCXMLExecutor) which is responsible for running with the configuration. For performance reasons, the configuration should only be created once, as it parses the XML document. In my demo, I instantiated it at class load time in the subclass of AbstractStateMachine.

So what do you do, if you are loading an object from a persistent store like a database and you want to instantiate your state machine (SCXMLExecutor) with a state other than the initial state? Well, it's quite easy, you simply use the API to modify the current state of the state machine (SCXMLExecutor) instance and set the initial state to be that, which your persisted entity currently has. The method isn't setCurrentState(), rather you use the current state object, and remove its states and set the relevant state which you retrieved from the configuration (SCXML). Have a look at AbstractStateMachine.setInitialState(SCXMLExecutor, String) to see how. Thanks to the SCXML team for showing me how to do this!

So having done all that, I was now in a position to use the state machine to manage the state in a persistent entity. I created a database table with a varchar column for holding my state, and primary key. In the real world, the state would be part of a much larger entity with other fields. I then setup my Eclipse project to contain the JPA facet and by right clicking on the project was able to generate my JPA entity classes from the table, such as shown in the SomethingPersistentWithState class. The only special thing I did, was to select "property based access" as the way in which JPA accesses the attributes. Normally, and by default, JPA uses field based access, i.e. it uses reflection at runtime to get and set the class attributes. By selecting "property based access", Eclipse generates an additional Annotation on my entity class: @Access(AccessType.PROPERTY). I then had only three things to do:

Change the "state" class attribute from a String into an instance of
StateMachineDemo, which is the implementation of AbstractStateMachine.
I can do this, because the above Annotation means that JPA doesn't care how I implement the
attribute, is will always use bean conform accessors to set/get the state of my entity, namely the
String getState() and void setState(String) methods.

Change the generated void setState(String) method visibility to become private, so
that no one could explicitly set the state of my entity. The method needs to exist, so that JPA can
access the attribute to persist it, but JPA can happily live with the method being private.

Add methods for each transition, which simply delegate the call to the state machine. Users of the
persistent entity use these trigger methods to change the state, rather than calling
void setState(String) on the entity.

These changes are all shown/documented in the StateMachineDemo class.

To demonstrate the state machine working, as well as it being used in a persistent environment, have a look at the JUnit test case StateMachineDemoTest. This class contains tests for setting any state during construction, tests for state transitions proving that only valid ones are allowed, as well as reading/writing an entity containing state to the database and updating the state along the way.

Eating a waffle, I just developed a new theorem. First assume the waffle is square and made up of little squares. If, like me, you eat around two edges, one square deep, so that the waffle stays square, you ALWAYS eat an odd number of squares. Eg. 6x6, you eat 6 mini squares, follwed by 5, making 11 (odd number), so that the waffle itself stays square, now only 5x5. My theorem hypothesises that this is true for all waffle sizes. In fact it's called the "Waffle Theorem", and here is the mathematical proof:

y = x^2 - (x-1)^2

-> y = x^2 - (x^2 - 2x + 1)
-> y = 2x-1.

QED, because 2 times anything ("2x") is always even, and subtracting one ("-1") always makes an even number odd. Hence my hypothesis holds true, regardless of waffle size.

Not bad for a Sunday night, all I wanted to do was eat the damn waffle... I have to say though, it's this kind of thing that I love about maths.