A few years ago I helped start what is now probably Austria's largest internet food delivery website. 'Mjam' is an onomatopoeia for the sound you make with your tongue when your mouth is watering in anticipation of something really tasty.

While working on Mjam, one thing we really thought about was ranking quality: How can we learn from our users and find out which restaurants are the best? The better we are at satisfying our customers, the more likely they'd stick around or even get their friends to use our site.

We immediately knew that Pure 1-5 Star ratings wouldn't do. Everyone uses stars differently, and there's interaction-specific attractors to boot (on Netflix people are more likely to rate movies they like, so 5 is more popular than 4, 4 is more popular than 3, ...). The Star rating system also loses a lot of resolution: We don't want to know which 50 restaurants have 5 Stars, we want to know which ones are the best. Ordering restaurants would be a much better fit than discrete rankings.

So we experimented with modeling the knowledge our users had when they made an order: A restaurant was an unknown to a user, unless they had recently placed an order there (within the last two months). If a user knew multiple restaurants, and clearly preferred a few of them over the others, that was a vote. Not ordering from a restaurant you hadn't tried before wasn't a vote against it, nor ordering from a restaurant without knowing any of the alternatives. We completely scrapped explicit votes. Instead we decided to let people vote with their money and real orders. The more a user knew (through previous orders), the more weight we gave to that user's behavior in calculating our rankings.

The method turned out to work out incredibly well: Mjam's ratings are still by far the best I've seen (I'm no longer a shareholder nor do I work with them or have any recent inside information). For newer projects I'm focusing even more on personalizing filters, but I think this kind of information modeling still has a lot of promise for improving a lot of ranking systems. Yelp would be a prime candidate for recommending you alternatives and then tracking over time which one you prefer.