How To Implement Learning Vector Quantization From Scratch With Python

A limitation of k-Nearest Neighbors is that you must keep a large database of training examples in order to make predictions.

The Learning Vector Quantization algorithm addresses this by learning a much smaller subset of patterns that best represent the training data.

In this tutorial, you will discover how to implement the Learning Vector Quantization algorithm from scratch with Python.

After completing this tutorial, you will know:

How to learn a set of codebook vectors from a training data set.

How to make predictions using learned codebook vectors.

How to apply Learning Vector Quantization to a real predictive modeling problem.

Let’s get started.

Update Jan/2017: Changed the calculation of fold_size in cross_validation_split() to always be an integer. Fixes issues with Python 3.

Update Aug/2018: Tested and updated to work with Python 3.6.

How To Implement Learning Vector Quantization From Scratch With PythonPhoto by Tony Faiola, some rights reserved.

Description

This section provides a brief introduction to the Learning Vector Quantization algorithm and the Ionosphere classification problem that we will use in this tutorial

Learning Vector Quantization

The Learning Vector Quantization (LVQ) algorithm is a lot like k-Nearest Neighbors.

Predictions are made by finding the best match among a library of patterns. The difference is that the library of patterns is learned from training data, rather than using the training patterns themselves.

The library of patterns are called codebook vectors and each pattern is called a codebook. The codebook vectors are initialized to randomly selected values from the training dataset. Then, over a number of epochs, they are adapted to best summarize the training data using a learning algorithm.

The learning algorithm shows one training record at a time, finds the best matching unit among the codebook vectors and moves it closer to the training record if they have the same class, or further away if they have different classes.

Once prepared, the codebook vectors are used to make predictions using the k-Nearest Neighbors algorithm where k=1.

The algorithm was developed for classification predictive modeling problems, but can be adapted for use with regression problems.

Ionosphere Dataset

The Ionosphere dataset predicts the structure of the ionosphere given radar return data.

Each instance describes the properties of radar returns from the atmosphere and the task is to predict whether or not there is structure in the ionosphere.

There are 351 instances and 34 numerical input variables, 17 pairs of 2 for each radar pulse that generally have the same scale of 0-1. The class value is a string with a value of either a “g” for good return or “b” for a bad return.

Using the Zero Rule Algorithm that predicts the class with the most observations, a baseline accuracy of 64.286% can be achieved.

Download the dataset and place it in your current working directory with the name ionosphere.csv.

Tutorial

This tutorial is broken down into 4 parts:

Euclidean Distance.

Best Matching Unit.

Training Codebook Vectors.

Ionosphere Case Study.

These steps will lay the foundation for implementing and applying the LVQ algorithm to your own predictive modeling problems.

1. Euclidean Distance

The first step needed is to calculate the distance between two rows in a dataset.

Rows of data are mostly made up of numbers and an easy way to calculate the distance between two rows or vectors of numbers is to draw a straight line. This makes sense in 2D or 3D and scales nicely to higher dimensions.

We can calculate the straight line distance between two vectors using the Euclidean distance measure. It is calculated as the square root of the sum of the squared differences between the two vectors.

1

distance = sqrt( sum( (x1_i - x2_i)^2 )

Where x1 is the first row of data, x2 is the second row of data and i is the index for a specific column as we sum across all columns.

With Euclidean distance, the smaller the value, the more similar two records will be. A value of 0 means that there is no difference between two records.

Below is a function named euclidean_distance() that implements this in Python.

1

2

3

4

5

6

# calculate the Euclidean distance between two vectors

def euclidean_distance(row1,row2):

distance=0.0

foriinrange(len(row1)-1):

distance+=(row1[i]-row2[i])**2

returnsqrt(distance)

You can see that the function assumes that the last column in each row is an output value which is ignored from the distance calculation.

We can test this distance function with a small contrived classification dataset. We will use this dataset a few times as we construct the elements needed for the LVQ algorithm.

1

2

3

4

5

6

7

8

9

10

11

X1 X2 Y

2.7810836 2.550537003 0

1.465489372 2.362125076 0

3.396561688 4.400293529 0

1.38807019 1.850220317 0

3.06407232 3.005305973 0

7.627531214 2.759262235 1

5.332441248 2.088626775 1

6.922596716 1.77106367 1

8.675418651 -0.242068655 1

7.673756466 3.508563011 1

Putting this all together, we can write a small example to test our distance function by printing the distance between the first row and all other rows. We would expect the distance between the first row and itself to be 0, a good thing to look out for.

The full example is listed below.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

from math import sqrt

# calculate the Euclidean distance between two vectors

def euclidean_distance(row1,row2):

distance=0.0

foriinrange(len(row1)-1):

distance+=(row1[i]-row2[i])**2

returnsqrt(distance)

# Test distance function

dataset=[[2.7810836,2.550537003,0],

[1.465489372,2.362125076,0],

[3.396561688,4.400293529,0],

[1.38807019,1.850220317,0],

[3.06407232,3.005305973,0],

[7.627531214,2.759262235,1],

[5.332441248,2.088626775,1],

[6.922596716,1.77106367,1],

[8.675418651,-0.242068655,1],

[7.673756466,3.508563011,1]]

row0=dataset[0]

forrow indataset:

distance=euclidean_distance(row0,row)

print(distance)

Running this example prints the distances between the first row and every row in the dataset, including itself.

1

2

3

4

5

6

7

8

9

10

0.0

1.32901739153

1.94946466557

1.55914393855

0.535628072194

4.85094018699

2.59283375995

4.21422704263

6.52240998823

4.98558538245

Now it is time to use the distance calculation to locate the best matching unit within a dataset.

2. Best Matching Unit

The Best Matching Unit or BMU is the codebook vector that is most similar to a new piece of data.

To locate the BMU for a new piece of data within a dataset we must first calculate the distance between each codebook to the new piece of data. We can do this using our distance function above.

Once distances are calculated, we must sort all of the codebooks by their distance to the new data. We can then return the first or most similar codebook vector.

We can do this by keeping track of the distance for each record in the dataset as a tuple, sort the list of tuples by the distance (in descending order) and then retrieve the BMU.

Below is a function named get_best_matching_unit() that implements this.

1

2

3

4

5

6

7

8

# Locate the best matching unit

def get_best_matching_unit(codebooks,test_row):

distances=list()

forcodebook incodebooks:

dist=euclidean_distance(codebook,test_row)

distances.append((codebook,dist))

distances.sort(key=lambda tup:tup[1])

returndistances[0][0]

You can see that the euclidean_distance() function developed in the previous step is used to calculate the distance between each codebook and the new test_row.

The list of codebook and distance tuples is sorted where a custom key is used ensuring that the second item in the tuple (tup[1]) is used in the sorting operation.

Finally, the top or most similar codebook vector is returned as the BMU.

We can test this function with the small contrived dataset prepared in the previous section.

The complete example is listed below.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

from math import sqrt

# calculate the Euclidean distance between two vectors

def euclidean_distance(row1,row2):

distance=0.0

foriinrange(len(row1)-1):

distance+=(row1[i]-row2[i])**2

returnsqrt(distance)

# Locate the best matching unit

def get_best_matching_unit(codebooks,test_row):

distances=list()

forcodebook incodebooks:

dist=euclidean_distance(codebook,test_row)

distances.append((codebook,dist))

distances.sort(key=lambda tup:tup[1])

returndistances[0][0]

# Test best matching unit function

dataset=[[2.7810836,2.550537003,0],

[1.465489372,2.362125076,0],

[3.396561688,4.400293529,0],

[1.38807019,1.850220317,0],

[3.06407232,3.005305973,0],

[7.627531214,2.759262235,1],

[5.332441248,2.088626775,1],

[6.922596716,1.77106367,1],

[8.675418651,-0.242068655,1],

[7.673756466,3.508563011,1]]

test_row=dataset[0]

bmu=get_best_matching_unit(dataset,test_row)

print(bmu)

Running this example prints the BMU in the dataset to the first record. As expected, the first record is the most similar to itself and is at the top of the list.

1

[2.7810836, 2.550537003, 0]

Make predictions with a set of codebook vectors is the same thing.

We use the 1-nearest neighbor algorithm. That is, for each new pattern we wish to make a prediction for, we locate the most similar codebook vector in the set and return its associated class value.

Now that we know how to get the best matching unit from a set of codebook vectors, we need to learn how to train them.

3. Training Codebook Vectors

The first step in training a set of codebook vectors is to initialize the set.

We can initialize it with patterns constructed from random features in the training dataset.

Below is a function named random_codebook() that implements this. Random input and output features are selected from the training data.

1

2

3

4

5

6

# Create a random codebook vector

def random_codebook(train):

n_records=len(train)

n_features=len(train[0])

codebook=[train[randrange(n_records)][i]foriinrange(n_features)]

returncodebook

After the codebook vectors are initialized to a random set, they must be adapted to best summarize the training data.

This is done iteratively.

Epochs: At the top level, the process is repeated for a fixed number of epochs or exposures of the training data.

Training Dataset: Within an epoch, each training pattern is used one at a time to update the set of codebook vectors.

Pattern Features: For a given training pattern, each feature of a best matching codebook vector is updated to move it closer or further away.

The best matching unit is found for each training pattern and only this best matching unit is updated. The difference between the training pattern and the BMU is calculated as the error. The class values (assumed to be the last value in the list) are compared. If they match, the error is added to the BMU to bring it closer to the training pattern, otherwise, it is subtracted to push it further away.

The amount that the BMU is adjusted is controlled by a learning rate. This is a weighting on the amount of change made to all BMUs. For example, a learning rate of 0.3 means that BMUs are only moved by 30% of the error or difference between training patterns and BMUs.

Further, the learning rate is adjusted so that it has maximum effect in the first epoch and less effect as training continues until it has a minimal effect in the final epoch. This is called a linear decay learning rate schedule and can also be used in artificial neural networks.

We can summarize this decay in learning rate by epoch number as follows:

1

rate = learning_rate * (1.0 - (epoch/total_epochs))

We can test this equation by assuming a learning rate of 0.3 and 10 epochs. The learning rate each epoch would be as follows:

1

2

3

4

5

6

7

8

9

10

11

Epoch Effective Learning Rate

0 0.3

1 0.27

2 0.24

3 0.21

4 0.18

5 0.15

6 0.12

7 0.09

8 0.06

9 0.03

We can put all of this together. Below is a function named train_codebooks() that implements the procedure for training a set of codebook vectors given a training dataset.

The function takes 3 additional arguments to the training dataset, the number of codebook vectors to create and train, the initial learning rate and the number of epochs for which to train the codebook vectors.

You can also see that the function keeps track of the sum squared error each epoch and prints a message showing the epoch number, effective learning rate and sum squared error score. This is helpful when debugging the training function or the specific configuration for a given prediction problem.

You can see the use of the random_codebook() to initialize the codebook vectors and the get_best_matching_unit() function to find the BMU for each training pattern within an epoch.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

# Train a set of codebook vectors

def train_codebooks(train,n_codebooks,lrate,epochs):

codebooks=[random_codebook(train)foriinrange(n_codebooks)]

forepoch inrange(epochs):

rate=lrate *(1.0-(epoch/float(epochs)))

sum_error=0.0

forrow intrain:

bmu=get_best_matching_unit(codebooks,row)

foriinrange(len(row)-1):

error=row[i]-bmu[i]

sum_error+=error**2

ifbmu[-1]==row[-1]:

bmu[i]+=rate *error

else:

bmu[i]-=rate *error

print('>epoch=%d, lrate=%.3f, error=%.3f'%(epoch,rate,sum_error))

returncodebooks

We can put this together with the examples above and learn a set of codebook vectors for our contrived dataset.

Below is the complete example.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

from math import sqrt

from random import randrange

from random import seed

# calculate the Euclidean distance between two vectors

def euclidean_distance(row1,row2):

distance=0.0

foriinrange(len(row1)-1):

distance+=(row1[i]-row2[i])**2

returnsqrt(distance)

# Locate the best matching unit

def get_best_matching_unit(codebooks,test_row):

distances=list()

forcodebook incodebooks:

dist=euclidean_distance(codebook,test_row)

distances.append((codebook,dist))

distances.sort(key=lambda tup:tup[1])

returndistances[0][0]

# Create a random codebook vector

def random_codebook(train):

n_records=len(train)

n_features=len(train[0])

codebook=[train[randrange(n_records)][i]foriinrange(n_features)]

returncodebook

# Train a set of codebook vectors

def train_codebooks(train,n_codebooks,lrate,epochs):

codebooks=[random_codebook(train)foriinrange(n_codebooks)]

forepoch inrange(epochs):

rate=lrate *(1.0-(epoch/float(epochs)))

sum_error=0.0

forrow intrain:

bmu=get_best_matching_unit(codebooks,row)

foriinrange(len(row)-1):

error=row[i]-bmu[i]

sum_error+=error**2

ifbmu[-1]==row[-1]:

bmu[i]+=rate *error

else:

bmu[i]-=rate *error

print('>epoch=%d, lrate=%.3f, error=%.3f'%(epoch,rate,sum_error))

returncodebooks

# Test the training function

seed(1)

dataset=[[2.7810836,2.550537003,0],

[1.465489372,2.362125076,0],

[3.396561688,4.400293529,0],

[1.38807019,1.850220317,0],

[3.06407232,3.005305973,0],

[7.627531214,2.759262235,1],

[5.332441248,2.088626775,1],

[6.922596716,1.77106367,1],

[8.675418651,-0.242068655,1],

[7.673756466,3.508563011,1]]

learn_rate=0.3

n_epochs=10

n_codebooks=2

codebooks=train_codebooks(dataset,n_codebooks,learn_rate,n_epochs)

print('Codebooks: %s'%codebooks)

Running this example trains a set of 2 codebook vectors for 10 epochs with an initial learning rate of 0.3. The details are printed each epoch and the set of 2 codebook vectors learned from the training data is displayed.

We can see that the changes to learning rate meet our expectations explored above for each epoch. We can also see that the sum squared error each epoch does continue to drop at the end of training and that there may be an opportunity to tune the example further to achieve less error.

Now that we know how to train a set of codebook vectors, let’s see how we can use this algorithm on a real dataset.

4. Ionosphere Case Study

In this section, we will apply the Learning Vector Quantization algorithm to the Ionosphere dataset.

The first step is to load the dataset and convert the loaded data to numbers that we can use with the Euclidean distance calculation. For this we will use the helper function load_csv() to load the file, str_column_to_float() to convert string numbers to floats and str_column_to_int() to convert the class column to integer values.

We will evaluate the algorithm using k-fold cross-validation with 5 folds. This means that 351/5=70.2 or just over 70 records will be in each fold. We will use the helper functions evaluate_algorithm() to evaluate the algorithm with cross-validation and accuracy_metric() to calculate the accuracy of predictions.

Extensions

This section lists extensions to the tutorial that you may wish to explore.

Tune Parameters. The parameters in the above example were not tuned, try different values to improve the classification accuracy.

Different Distance Measures. Experiment with different distance measures such as Manhattan distance and Minkowski distance.

Multiple-Pass LVQ. The codebook vectors may be updated by multiple training runs. Experiment by training with large learning rates followed by a large number of epochs with smaller learning rates to fine tune the codebooks.

Update More BMUs. Experiment with selecting more than one BMU when training and pushing and pulling them away from the training data.

More Problems. Apply LVQ to more classification problems on the UCI Machine Learning Repository.

Did you explore any of these extensions?
Share your experiences in the comments below.

Review

In this tutorial, you discovered how to implement the learning vector quantization algorithm from scratch in Python.

Specifically, you learned:

How to calculate the distance between patterns and locate the best matching unit.

How to train a set of codebook vectors to best summarize the training dataset.

How to apply the learning vector quantization algorithm to a real predictive modeling problem.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

I have used the LVQ on an arff file and I am getting very bad performance from the algorithm. The accuracy is around 60%. Can you try to use the algorithm on this dataset and let me know your results or ways to improve it?

I wanted to know if we can use LVQ for instance selection. For e.g. if there are 15000 instances in my dataset and I only want to use 5000 of those instances as the training dataset. There are various ‘Instance Selection’ algorithms and I read that we can LVQ as well. But I am not sure how can I do that using the example that you have given above.

Correct me if i am wrong.
What i understand from the question asked by curious_kid is :
Say we are given a dataset of 15000 instances..
example
0.000133,5,0.446809,0.089798,0.516662,0.003467,0.422915,0.414912,UP
0.000133,5,0.468085,0.047676,0.50476,0.003467,0.422915,0.414912,DOWN
0.000133,5,0.489362,0.146902,0.495537,0.003467,0.422915,0.414912,UP
0.000133,5,0.510638,0.193287,0.485272,0.003467,0.422915,0.414912,UP
.
.
.
………

Now he wants to use LVQ for instance selection(data pre processing). That means that the final result should have 5000 best instances which he can later use for classification.

Now as suggested by you, it could be done by taking some threshold and only pick instances within a distance of codebook vectors.

Doing this means that we are picking instances from the dataset on the basis of distance from the random codebooks generated. For e.g. in the function “get_best_matching_unit”, if we use this distance “distances.append((codebook, dist))”(line 86), then we won’t be training the codebooks as done in your code in the lines 109-114 and hence the main functionality of LVQ won’t be implemented(i.e. to train the random codebooks over epochs and adjusting bmu according to the error). If we do train the codebooks, then we will lose the distance values, because you are returning “distances[0][0]” in line 88.

The way I am thinking to approach this problem is by training the codebooks as done by you in the code and getting the final value of codebooks as done in line 115. Giving the value of n_codebooks as 5000, which will give us the best 5000 instances representing the data ( as one codebook represent an instance). And then use those 5000 instances for classification(using any classification algorithm).

Do you have any suggestions for this approach? I would be glad to know about it.

Interesting. Using LVQ this way is performing a type of outlier removal.

It will probably be quite effective at lower dimensions (in data without outlier observations) and fall apart at higher dimensions given the curse of dimensionality as distances and thresholds quickly become less meaningful. Generally LVQ and kNN breakdown at higher dimensions for the same reason:https://en.wikipedia.org/wiki/Curse_of_dimensionality

Yes, the codebook vectors could be taken as centroids or a projection of the data. I would contrast this approach with projection methods like PCA, sammons mapping, tsne and others. Develop models from each projection and compare the performance on resulting models in a hold out dataset.

I am thinking of using the euclidean distance between the codebook vectors and the new data and then calculate the mean of those distances which can act as the threshold. Then I am planning to select all those instances for which the euclidean distance is less than the calculated mean value. Does that sound good to you or do I need to think in a different way?

I could not understand the functionality of “learning_vector_quantization” function. For e.g. where is it called in the code? And thus the “predict” function will not be called. I am getting the same result by commenting the predict function. Please correct me if I am wrong or if I am missing something here.

Thank you Jason. I understood it now. I have one more doubt. You have mentioned in your blog that once the codebook vectors are prepared, those codebook vectors are used to make predictions using 1-NN. But I am not able to understand that how are you doing it in your code. Please elaborate it a bit.

I’ve created a LVQ network with trained data and two samples. It works, but how would I go about getting confidence values for untrained data? Ex. confidence of .98 means the sample is really close to the actual model of one of my samples.

Thanks for your reply. As I normalize all the values between 0 and 1, what would I use for a maximum? I tried using 1 – distance(bestUnitMatch, sample) and I get values of .96; and if that’s right I’m so amazed because I only trained the network with 30 samples for each 2 of the sets

In the “train_codebooks()” function where is the “codebooks” variable getting updated?.Only the variable “bmu” is moved towards or away from the training vector which is not used to update the “codebooks” vector.

Hi, Thank you for this lovely illustration. I am not a pro in python but have some questions.
On line 61, train_set.remove(fold) returns a funny error “ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()”
Is there any alternative around this. ?

Also i was wondering the reason for using lists and not numpy array like dataframe. Any particular reason?