Introduction

I'd like to present a console based implementation of the backpropogation neural network C++ library I developed and used during my research in medical data classification and the CV library for face detection: Face Detection C++ library with Skin and Motion analysis. There are some good articles already present at The CodeProject, and you may consult them for the theory. In my code, I present the necessary features as input data preprocessing in the input layer with Minmax, Zscore, Sigmoidal, and Energy normalization. These parameters are obtained from the training set, and then used for preprocessing every incoming vector for classification. The console supports training data random separation to train, validation, and test sets before backpropagation training. Random separation allows to obtain a representative train set comparing performance on validation and test parts. A validation set is useful for preventing over-fitting by estimating the performance on that set. At the end of the backpropagation session, I save both network configurations, the one with the best performance on the validation set and the last training epoch configuration. For performance estimation, I use sensitivity, specificity, positive predictivity, negative predictivity, and accuracy metrics. For validation performance estimation in the case of biased data distribution (for example, in my Face Detection article, you may find that there are a lot more non-faces than faces, 19:1 ratio) I provide a geometric mean and F-measure metrics to support the scenario. To support a large amount of data vectors, I provide File Mapping based data load. That allows to map your hundreds of megs of training data to memory in 1 msec and start your training session immediately. For relatively small amounts of data, you may use a text file format. Finally, the console implementation is easier to use, you avoid a lot of mouse clicking in GUI applications, and may automate the process with batch files for choosing the right network topology, the best performance on the validation and test sets, and so on.

It will use the network.nn file as a neural network, and load data form data1_file and data2_file, which represents data vectors from positive and negative classes, and train it for 1000 epochs.

The neural network file format is described in my Face Detection article. To start with random initialized weights before the training session, you need to provide only the number of layers and the number of neurons per layer in that file. For example, in the demo zip, you will find a iris.nn file:

3
4 8 1

Three layers, and 4, 8, and 1 neurons per layer.

It supports two data file formats. The text one is:

vector_name class_mark
x1 x2 x3 ... xN
...

vector_name is the name of your particular vector; it should not start with a numeric element, only with a letter. class_mark is the non zero number corresponding to the class attribute: 1, 2, etc. In the console application, I use only 1 as positive class with a 0.9 desired output, and 2 as negative with 0.1 as the desired output. The next line contains the vector entries in integer or floating point format.

In the demo zip, the IRIS data is organized using a text format with four-dimensional entries:

The binary floating point file format is expedient when you have a large amount of data. The data is saved in a separate file as a sequence of floating point numbers in binary format, using 4 bytes per floating point number:

file1.dat (2x3 matrix)

[x11] [x12] [x13] [x21] [x22] [x23]

And, the dimensions of the data matrix are saved in the file with the same name but with a .hea extension.

file1.hea

2 3

In the previous example, the file file1.dat contains two three-dimensional vectors.

In that case, your data1_file or data2_file may contain the entries with the full path to the files and the class marks:

fullpath\file1.dat 1
fullpath\file2.dat 1
...

The next parameters to the console application for backprop training are optional. You may use them for validation and testing of your network, for input data normalization, and error limits during training process.

This command line demonstrates that you use your validation and test sets in the vld_file and tst_file files in text or binary format, as described above, with a validation threshold 0.5 (that is, the network output greater than 0.5 attributes a data vector to a positive class), with geometric mean of sensitivity and specificity as the performance metric for validation stopping and Zscore normalization. The allowed validation metrics are specified at the end of the console help line. If your vld_file or tst_file are empty files, then the corresponding data set will be composed of randomly selected entries from your training set, with 25% records from each class.

The last eleventh argument to the console is the error tolerance. If the difference between the desired output and the mean network output for positive and negative classes is less than the error for 10 consecutive epochs, the training stops. During the backpropagation training, if the difference between the desired output and the network output for a particular vector is less than the error you specify, the network weights are not adjusted. This allows to correct the network weight connections for the data vectors which are not yet 'memorized'.

The next example demonstrates the sample training session for the IRIS data:

The network configuration is saved to maxacur.nn corresponding to the best performance on the validation set, and the last epoch configuration is saved to iris.nn. At the end, you may compare the results for them.

To use your trained networks for testing, just run the console with these parameters:

>ann1dn r iris.nn test_data

The test_data file is in text or binary format. You may provide as the fourth argument the threshold (default is 0.5) to obtain the ROC curve on your test set, for example, varying it between 0.0 and 1.0.

Neural Network Classes

The neural network is composed from the following classes:

ANNetwork

ANNLayer

ANeuron

ANLink

The ANNetwork class contains the implementation of the neural network for users of the library. To avoid protected interface programming for the rest of the classes, I used friends. I'll describe the library structure first, and then provide the functions you need to use from the ANNetwork class to maintain your own implementations.

The ANNetwork contains an array of ANNLayer layers. Every layer contains an array of ANeuron neuron objects, and every neuron contains arrays of ANLink input and output connections. With that design, you may arrange any desired network structure; however, in my implementation, I provide only feed-forward full connectionist structure.

The basic unit of the neural network is the neuron class, ANeuron. You may add bias or input connection to it, represented as the ANLink object.

void ANeuron::add_bias()

void ANeuron::add_input(ANeuron *poutn)

The bias connection always take 1.0f as an input value, as you know. With add_input(), you add connection to the neuron, supplying with its argument the neuron from the previous layer to which it connects.

So, every neuron 'knows' which neurons from the next layer connect to its output. The ANLink is like the 'arrow', pointing from the neuron in the previous layer, ANLink::poutput_neuron to the neuron in the next layer, ANLink::pinput_neuron.

ivec and ivec represent the input vector fed to a neural network and the output vector where it stores its results. Their dimensions should match the number of input and output neurons in the network structure. dsrdvec is the desired output vector to which it adjusts its connections to match it within the error tolerance. The ANNetwork::train() function will return true in case the backpropogation took place, or false if the network output was within error to the desired vector.

I am new guys in this area and spent days reading you article. They are great implementation codes and I am interesed in it.

In this article, "ann1dn t iris.nn setosa_versi.dat virgi.dat" generate iris.nn file. How could I generate the setosa_versi.dat and virgi.dat? Could you let me know how to generate the "setosa_versi.dat" and "virgi.dat" file from face images?

I have a problem with class mark.In your exemple, you put only class 1 or 2 to define a positive(0.9 output desired) or negative(0.1 output desired) class.In my data training I don't have only positive or negative output.How can I put more output desired (different to true or false)? Is it possible ?

That console app on codeproject is limited to training on 1 output neuron only. My ANN lib supports any number of output neurons 3, 10, 100 ... You need just to modify main console file to feed to ANN lib desired outputs like:0.9 0.1 0.1 for 1st class0.1 0.9 0.1 for 2st class0.1 0.1 0.9 for 3st classand so on, for any number of output neurons

you have to put cls1, cls2, cls3, ... entries as you presented in your post to one file cls and provide just empty file like void and do the training:>>ann1dn.exe t net.nn cls void 1000

I've got developed before the same looking console that induces ANN to train and run on any number of outputs as I described, I can give you it but not for free