My trajectory through “Machine Learning in ActionMachine Learning in Action” is becoming more unpredictable as we go – this time, rather than completing our last episode on K-means clustering (we’ll get back to it later), I’ll make another jump directly to Chapter 14, which is dedicated to Singular Value Decomposition, and convert the example from Python to F#.

The chapter illustrates how Singular Value Decomposition (or SVD in short) can be used to build a collaborative recommendation engine. We will follow the chapter pretty closely: today we will focus on the mechanics of using SVD in F# – and leave the recommendation part to our next installment.

Until this point, I have avoided using a Linear Algebra library, because the algorithms we discussed so far involved lightweight, row-centric operations, which didn’t warrant taking such a dependency. SVD is one of these cases where using an established library is a good idea, if only because implementing it yourself would not be trivial. So let’s create a new script file (Chapter14.fsx), add a reference to Math.NET Numerics for F# to our project via NuGet, and reference it in our script:

Now that we have our tools, let’s start working our example. Imagine that we are running a website, where our users can rate dishes, from 1 (horrendous) to 5 (delightful). Our data would look something along these lines:

Our goal will be to provide recommendations to User for Dishes they haven’t tasted yet, based on their ratings and what other users are saying.

Our first step will be to represent this as a Matrix, where each Row is a User, each Column a Dish, and the corresponding cell is the User Rating for that Dish. Note that not every Dish has been rated by every User – we will represent missing ratings as zeroes in our matrix:

We initialize our 11 x 11 matrix, which creates a zero-filled matrix, and then map our user ratings to each “cell”. Because we constructed our example that way, our UserIds go from 0 to 10, and DishIds from 0 to 10, so we can map them respectively to Rows and Columns.

Note: while this sounded like a perfect case to use a Sparse Matrix, I chose to go first with a DenseMatrix, which is more standard. I may look at whether there is a benefit to going sparse later.

Note: our matrix happens to be square, but this isn’t a requirement.

Note: I will happily follow along the book author and replace unknown ratings by zero, because it’s very convenient. I don’t fully get how this is justified, but it seems to work, so I’ll temporarily suspend disbelief and play along.

At that point, we have our data matrix ready. Before going any further, let’s write a quick utility function, to “pretty-render” matrices:

We iterate over each row and column, start a newline every time we hit column 0, and print every value, nicely formatted with 2 digits after the decimal.

In passing, note the F#-friendly Matrix.iteri syntax – the good people at Math.NET do support F#, and MathNet.Numerics.FSharp.dll contains handy helpers, which allow for a much more functional usage of the library. Thanks, guys!

Now is the moment when I wave my hands in the air, and say “let’s run a Singular Value Decomposition”. I won’t even attempt to explain how or why it works, because this would be way beyond the scope of a single post (and to be perfectly honest, because my linear algebra is pretty rusty). Rather, I’ll do it, and I hope that the results will convey some of the magic that it happening:

The Singular Value Decomposition breaks a matrix into the product of 3 matrices U, Sigma and VT. We call the SVD procedure on our data matrix, and retrieve these 3 elements from the result: U and VT, which are both already in matrix form, and sigma, a vector listing the Singular Values, from which we recompose the expected S diagonal matrix, using the vector elements to populate the diagonal.

Great. Now instead of one matrix, we have three – what’s so special about U, S and VT?

First, we have by definition of the SVD, data = U x S x Vt. Let’s check that this holds:

Note that the values on the diagonal are in decreasing order. One way to describe what is going on is that SVD reorganizes the matrix data in a more efficient manner, extracting “concepts/categories” from the matrix, and, paraphrasing the Wikipedia article on Latent Semantic Indexing, U can be seen as a User-to-Category matrix and Vt as a Category-to-Dish matrix – and the Singular Values in between represent the “importance” of each extracted categories.

Let’s see if we can illustrate this. First, let’s compute the “user-to-category” matrix, U x S:

letuserToCategory=U*S|>pretty

We get the following, where rows map to users, and columns to the extracted “categories”:

This suggests that users with Id 3, 4 and 6 (highlighted values in rows 4, 5 and 7) are strongly tied to Category 1 (the first column). Looking back at the original data matrix, we can see that all 3 gave high ratings to the first and third dish, and two of them rated the second dish high as well. So we would expect Category 1 to map to “liked dish 1, 2 and 3”.

Sure enough, the first row, which maps to our first Category, shows 3 high amplitude values (highlighted), in the first three columns. Category 1 could be described as “like Dish 1 and Dish 3, and also to some extent Dish 2”.

To summarize, what the SVD gave us is a reorganization of our data, restating the original matrix in terms of “Categories”, and transforming the feature space into a more “informative” one, mapping users to dishes through categories which combine features and have an associated strength, represented by the singular values.

One way this can be exploited is to simplify data: the larger Singular Values are responsible for most of the shape of our matrix (the “important” categories), so we could drop the smaller ones without losing too much information.

Let’s see this in action – suppose we kept only a subset of the singular values, and dropped the smallest ones. We could simply set the diagonal value in the S matrix to 0, but, as the net effect of multiplying the matrices together will be to produce rows and columns of zeroes where we have a 0 value in the diagonal, we might as well drop these rows and columns from S altogether, and to maintain dimension consistency, drop the corresponding rows/columns from U and Vt. That way, we work with smaller matrices – winning!

The modification can be done like this – we drop the last columns of U, the last rows and columns of S, and the last rows of Vt, and we are set:

The largest value contributes to 36% of the energy, and the 5 first ones together are responsible for 90% of the shape of our matrix. Let’s see how an approximation using only 5 values looks like, by setting the value of **subset** to 5:

Obviously, this is not quite as close to the original matrix as before, but it’s still pretty good – and that, in spite of the fact that we just dropped 6 out of our 11 features. Not bad! Instead of storing 11 x 11 = 121 values, we now need only 11 x 5 + 11 x 5 + 5 = 115 values (we only need to store the diagonal values of S, because the rest is 0).

Of course, with 11 dishes and 11 users, it’s hardly worth the effort. However, imagine that we had 1,000 users, and that 5 singular values was still the magic number. In that case, our data matrix would be 1,000 x 11 = 11,000 values, whereas the reduced version would require 1,000 x 5 + 11 x 5 + 5 = 5,060 values, only 46% of the initial matrix. In this case, being able to reduce the set of features suddenly becomes a much more interesting proposition.

I’ll stop here for today - I hope this post conveyed first that Linear Algebra with Math.NET and F# is pretty easy, and then a sense for what Singular Value Decomposition does. Next time, we’ll look at how we could go about providing recommendations to users by analyzing the similarities between dish ratings – and then how we can take advantage of the SVD decomposition to reduce the features into a simplified representation and improve the process.