Introduction

In this article, I will show you how to create a feedforward neural network. I will show you how to create the feedforward neural network to recognize the XOR operator. There are many examples that will show you how to do this on the Internet. However, I am going to show you how to do this three times. We will use three different neural network frameworks. This all allows you to get a feel for the three major Java open source neural networks that are available on the Internet and how they approach things. This may help you to decide which neural network framework is best suited to your project.

First, let me explain the XOR operation that I will be teaching the neural network to recognize. The XOR operation is essentially the “Hello World” of neural networks. Neural networks accept input and produce output. They process data using layers of neurons. A feedforward neural network has an input layer, zero or more hidden layers, and an output layer. The input accepts an array of floating point numbers. The output layer produces an array of floating point numbers. The size of these arrays does not need to be equal. The number of neurons in the input and output layers determines the array size.

You can then create training data. For the XOR operator, you would have four training items. The training items map the input to the ideal output. For XOR, this training data would be as follows:

[0,0] -> [0]
[1,0] -> [1]
[0,1] -> [1]
[1,1] -> [0]

These values correspond to the mathematical XOR operator. The neural network will train until the output from the neural network is close to the ideal output. The training is broken up into iterations. Each iteration will get the neural network output closer to the ideal output. The difference between the actual output from the neural network and the ideal output is the error rate of the neural network. There are different ways to train a neural network, but they all usually involve going through iterations, until the error decreases to an acceptable level.

This article will show how to implement the XOR operator in three different Java open source neural network frameworks. The three main open source neural network frameworks that are available on the Internet are:

Creating the XOR Operator in Encog

Encog is licensed under the Lesser GNU Public License. Encog supports a variety of training techniques. In this article, we will make use of a training technique called Levenberg Marquardt, or LMA. LMA is one of the most advanced training techniques available for neural networks. It does not work well with all training sets, but it can learn the XOR in a fraction of the time other training techniques require. LMA requires that the output of the neural network be a single neuron. This is fine for the XOR operator. Also, as the size of the training set increases, the effectiveness of LMA decreases. For larger training sets, you might want to consider Resilient Propagation or Resilient Propagation.

First, we setup the training data. The XOR data is stored in two arrays.

Levenberg Marquardt training sometimes fails to converge from the random weights. This is especially true for such a small network as this. We will use a reset strategy to rerandomize if there is less than a 1% improvement over 5 iterations.

train.addStrategy(new RequiredImprovementStrategy(5));

We are now ready to train though the epochs, or iterations. We begin with epoch 1.

int epoch = 1;

We will loop through training iterations until the error falls below 1%.

As you can see, it took Encog 18 iterations with Levenberg Marquardt training. Not all training techniques will use as few iterations as Levenberg Marquardt. We will see this in the next section.

Creating the XOR Operator in Neuroph

Neuroph is another neural network framework. It is licensed under the Apache License. It is currently in discussions to be merged into the Apache Machine Learning project. For Neuroph, we will use an automatic variant of the backpropagation algorithm they provide. It is not nearly as advanced as Levenberg Marquard, and as a result will take considerably more training iterations.

As you can see, it took 613 iterations to get the error below 1%. Clearly backpropagation is not nearly as robust as Levenberg Marquard. Automatic backpropagation is currently the most advanced training offered by Neuroph. This is one of the areas where Encog really excels in. Encog offers advanced training methods such as Resilient Propagation, Simulated Annealing, Genetic Training, Levenberg Marquard and Scaled Conjugate Gradient.

However, the automatic additions made to backpropagation by Neuroph does give it an advantage over a neural network framework that just uses standard backpropagation.

Creating the XOR Operator in JOONE

JOONE is the oldest neural network framework that we are examining. Encog and Neuroph were both started in 2008. JOONE dates back to 2001. JOONE is not actively supported, though there are a number of systems that were implemented with it. JOONE makes use of backpropagation. JOONE has classes that claim support for resilient propagation, but I was unable to get it to work. JOONE is also known for being “buggy”, and because it is no longer an active project, this can make things difficult.

The last two frameworks that we examined had fairly simple APIs that make it easy on the programmer to get the neural network created. Not so with JOONE. JOONE’s API is a beast. It does not really hide much, and is very difficult to get working. Of all three examples, this one took me the most time. It is also the longest.

Here is one feature I really dislike about JOONE. In both Neuroph and Encog, you create a neural network and then train it. With JOONE, you must physically change the network to switch between training it and using it. Here we are setting up to train. We will use a memory synapse to feed the arrays of XOR into the network.

this.monitor = nnet.getMonitor();
monitor.setTrainingPatterns(4); // # of rows (patterns) contained in the input file
monitor.setTotCicles(1000); // How many times the net must be trained on the input patterns
monitor.setLearningRate(0.7);
monitor.setMomentum(0.3);
monitor.setLearning(true); // The net must be trained
monitor.setSingleThreadMode(true); // Set to false for multi-thread mode
/* The application registers itself as monitor's listener so it can receive
the notifications of termination from the net. */
monitor.addNeuralNetListener(this);
}

As you can see, it took almost 5000 iterations, and JOONE has not yet trained as well as Neuroph. This is because it is using regular backpropagation. The automatic propagation used by Neuroph varies the learning rate and threshold to optimize the learning process. The difference is abundantly clear here.

Conclusions

In this article, I showed you three different approaches to the same problem. In my opinion, Encog and Neuroph are easy choices over JOONE. JOONE is not really supported any longer, and though JOONE can execute an iteration much faster than Neuroph, Neuroph’s auto training makes it superior to JOONE. Also JOONE is very hard to use.

Encog has many more features than Neuroph and supports very advanced training methods. It took Encog 18 iterations, Neuroph 613 and JOONE over 5000. I find the internal code of Neuroph easier to follow than Encog. Though I find the APIs of Encog and Neuroph to be comparable. Encog is built for speed, and it makes its internals more complex. Encog can even take advantage of your GPU for additional training speed.

In my next article, I will benchmark all three and show timings in various situations.

This was very useful for me and helped me realize a lot of things, yet if u could recommend a reference to understand neural networks itself (I mean what its uses, training algorithm, .... etc) or should I search for such articles and it'll be enough ?
and again Thanks a lot

At the End nothing is difficult enough to stop me
let's be an exception