Filling up a matrix without doubles

I'm working on an item-based collaborative filter, for which you need similarities between items. I have found that create a matrix like below works well, but I now want to prevent it from calculating doubles.

I'll explain what I mean with an example. Assume you have a list of 5 items, each with ratings to base the similarity on.

After calculating similarities, I have come up with the following matrix: