A collection of sloppy snippets for scientific computing and data visualization in Python.

Sunday, September 15, 2013

Self Organizing Maps

The Self Organizing Maps (SOM), also known as Kohonen maps, are a type of Artificial Neural Networks able to convert complex, nonlinear statistical relationships between high-dimensional data items into simple geometric relationships on a low-dimensional display.
In a SOM the neurons are organized in a bidimensional lattice and each neuron is fully connected to all the source nodes in the input layer. An illustration of the SOM by Haykin (1999) is the following

Each neuron n has a vector wn of weights associated.
The process for training a SOM involves stepping through several training iteration until the item in your dataset are learnt by the SOM. For each pattern x one neuron n will "win" (which means that wn is the weights vector more similar to x) and this winning neuron will have its weights adjusted so that it will have a stronger response to the input the next time it sees it (which means that the distance between x and wn will be smaller). As different neurons win for different patterns, their ability to recognize that particular pattern will increase.
The training algorithm can be summarized as follows:

Initialize the weights of each neuron.

Initialize t = 0

Randomly pick an input x from the dataset

Determine the winning neuron i as the neuron such that

Adapt the weights of each neuron n according to the following rule

Increment t by 1

if t < tmax go to step 3

We have that η(t) is called learning rate and that h(i) is called neighborhood function which has high values for i and the neurons close to i on the lattice (a Gaussian centered on i is a good example of neighborhood function). And, when t increases η also decrease and h decrease its spread. This way at each training step the weights of the neurons close to the winning one are adjusted to have a stronger response to the current pattern. After the training process, we have that the locations of the neurons become ordered and a meaningful coordinate system for the input features is created on the lattice. So, if we consider the coordinates of the associated winning neuron for each patter the SOM forms a topographic map of the input patterns.

MiniSom is a minimalistic and Numpy based implementation of the SOM. I made it during the experiments for my thesis in order to have fully hackable SOM algorithm and lately I decided to release it on GitHub.
The next part of this post will show how to train MiniSom on the Iris Dataset and how to visualize the result. The first step is to import and normalize the data:

The snippet above reads the dataset from a CSV and creates a matrix where each row corresponds to a pattern. In this case, we have that each pattern has 4 dimensions. (Note that only the first 4 columns of the file are used because the fifth column contains the labels). The training process can be started as follows:

Now we have a 7-by-7 SOM trained on our dataset. MiniSom uses a Gaussian as neighborhood function and its initial spread is specified with the parameter sigma. While with the parameter learning_rate we can specify the initial learning rate. The training algorithm implemented decreases both parameters as training progresses. This allows rapid initial training of the neural network that is then "fine tuned" as training progresses.
To visualize the result of the training we can plot the average distance map of the weights on the map and the coordinates of the associated winning neuron for each patter:

For each pattern in the dataset the corresponding winning neuron have been marked. Each type of marker represents a class of the iris data ( the classes are setosa, versicolor and virginica and they are respectively represented with red, green and blue colors). The average distance map of the weights is used as background (the values are showed in the colorbar on the right).
As expected from previous studies on this dataset, the patterns are grouped according to the class they belong and a small fraction of Iris virginica is mixed with Iris versicolor.

For a more detailed explanation of the SOM algorithm you can look at its inventor's paper.

Hi Guntur, MiniSom has the attribute weights, it's a numpy matrix that can be save with the function numpy.save. If you want to reuse a MiniSom, you can crate an instance, train and save the weights. When you want to reuse that network, you can create a new instance, load the weights from a file and overwrite the attribute weights.

How does the SOM react to categorical inputs, e.g. [1,2,3,..] are categorized peoples favorite foods? Is there a way to work with this type of data? Does the training assume an ordinal number set or does it learn this on its own?

The algorithm assumes that the order of the values has a meaning. I suggest you to try a binary encoder like this one: http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html#sklearn.preprocessing.LabelBinarizer

helloI am using your MiniSom for anomaly detection in high dimentional data but I have a problem with the choice of the input parameters So how can we chose these parameters using the dimension of the input data?

Hi, I am using your MiniSom for abnormaly detection in a data stocked in Elasticsearch, so how to recover my data in your code ? and how to replace your iris.csv with my data? should i convert my data to numpy matrix? if yes how?

Fantastic piece of code, thanks for sharing it. I've been playing with it to classify oceanography data (pacific ocean temperatures) and am wondering about reproducibility. While it picks up the obvious patterns (i.e. El Nino) it seems that each time I run the algorithm I get different maps, sometimes patterns a show up as maps and other times are not present, and increasing the number of iterations doesn't seem to cause a convergence to consistent results. This is also true for your supplied iris and colors examples. Am I missing something, or do SOMs truly provide different results each time the algorithm is run?

Hi Eric, the result of the algorithm strongly depends on the initialization of the weights and they are randomly initialized. To have always the same results you can pass the argument random_seed to the constructor of MiniSom. If the seed is always the same, the weights will be initialized always the same way., hence you'll have always the same results.

Nice work.I have gone through your code,and I just wonder that why 'som.random_weights_init(data)' is called,for the weights already have been initialized in the MiniSom __init__ function。And 'som.random_weights_init(data)' regards normaliztion of the data as weights,is it a correct way?Also 'som.random_weights_init(data)' replaced the initial weights.

Because that initialization seemed more suitable in that case. The constructor initializes the weights using np.random.rand by default. It's totally fine to choose a different initialization if the dafault one doesn't suits you.

Hi, thank for answer my question. I have a doubt. I was testing your code with the mnist data set, is similar to the digits dataset from python, but the difference is the size of the images. I trained the network SOM with a sample of 225 random digits and the dimension of my grid is 15*15. When I plot the U-matrix with the method distance_map, each coordinate of my plot should have a digit rigth? Why the position of my winning neuron is the same for different samples? There are empty positions in muy plot wuth any digits. When training the SOM with a larger sample there are empty positions too in muy plot. Should that happen? I use other implementation for example kohonen library from R package does not happen. Could you help me understand why this happens?. Thanks

- If your samples represent digits, you can associate a cell in the map with a digit. See this example https://github.com/JustGlowing/minisom/blob/master/examples/example_digits.py

- If two samples have the same winning neuron, it means that the two samples are similar.

- With the training and initialization methods implemented, It's normal that some areas don't have winning neurons for the samples used for training, especially between regions that activate for samples that have different.

Thank you so much for providing the code. I hope you can help me with my question. I thought that it is possible to apply SOM for outlier detection in an unsupervised manner (without labeled data). In the iris dataset the data has labels right:

Hi, that's a great library you have implemented. I would like to try and combine a self organising map and a multilayer perceptron. I have used your network to cluster character images. Is there any way to save the clustering so I can feed it as input to a multilayer perceptron?

Hi, thankyou fot the fast response. I shall try this, thank you.I have labels associated with each image in the dataset very similar to t in your example. I want to associate the winner with a label if possible. When you use a for loop and use the enumerate function on the data, is the code t[cnt] associating your t label with the winner?

When we find the outliers on the map by looking at the highest distance (close to 1), how can we know to which observation in the original data it corresponds ? In other words, is there an inverse function of the winner() function to reverse the mapping from the input space to the output map ?

Thanks. The reason why I am asking this is that the clustering results and quantization errors are different from the results I got from MATLAB somtoolbox. After reviewing all the codes inside, I think the possible reason maybe the parameter settings on the training stage. After all, I think you did a brilliant job. Thanks!

I am having a trouble with your pylab code from your example digits.py which I found here: https://github.com/JustGlowing/minisom/blob/master/examples/example_digits.py

I have digits images I want to plot but the problem is a window titled figure 1 comes up with an axis all by itself and a separate window titled figure 2 pops up with the clustered images but without an axis. I tried to combine the 2 figures by removing figure(2) from the code and even though only one window titled figure 1 pops up there is no axis, just the clusteted images alone. On top of that not all of the clustered images are shown up. Do you know of anyone else having this problem who fixed it or if there is a way to combine the 2 figures or a way for it the axis and clustered images to show up using 1 figure?

I am trying to implement MiniSom for a 3D array. The array´s shape is [time(days),latitude,longitude] and the intention is to catalogue each day in one of four nodes. For the moment I am reshaping it to an array with the shape [time(days),len(latitude)*len(longitude)]. This way:

I would like to keep the original shape, but I do not know how to implement with miniSom. In the documentation (https://github.com/JustGlowing/minisom) is indicated that "you need your data organized as a Numpy matrix where each row corresponds to an observation or an as list of lists like the following", but my problem is that I have a 2-D array instead of a list/row. Therefore I do not know what to introduce as "input_len" in the class MiniSom or if train_random assimilates 3-D arrays...

Is there a way to get the coordinate of the grid by seting a value range of the colorbar without looking at the map? For example, if I want to get the coordinate of the grid with the value between 0.9 and 1.0 on the colorbar.

Hi, sigma is the initial spread of the neighborhood function. In case sigma is very small, only a neuron (the winning one) will be trained at each iteration. In case it's very big all the neurons will be trained.

Thank you very much for your code.Could you please tell me if is there any way to identify the occurrence of topological distortions (when the network folds over itself)?Like in this link: https://www.researchgate.net/figure/SOM-output-lattice-topology-distortion-examples-related-to-edge-folding-top-left_fig15_304024726

Thanks for the quick reply. I used the same, but it is only creating the single cluster (including all tokens in single cluster). Am I making mistake anywhere? Can you please share your thoughts on the same?

Yes, the data is normalized. Steps I followed is that I vectorized the tokens, used train_batch to train SOM and trying to visualize. Am I making a mistake in visualization? Or can you help me on how can I get the number of clusters created by SOM or visualize the same. It will be really helpful. Thanks.

Hi, firstly thanks for this great work. I have plotted the results of the SOM using the distance_map function. However, I now want to cluster similar neurons together. Am I correct in thinking that the distance_map represents the average distance of each neuron to its neighbourhood? Is there a way to get actual distances of each neuron so I could cluster neurons close to each other? Thanks