Because of this, a plugin that only one or two 5 star ratings would ranker higher (assuming all other things are similar) than a plugin that has a collective 4.5 stars from 50 ratings.

I would like to suggest that the number of ratings be used alongside the overall rating, where a higher number of ratings would have more impact.

There should be a threshold though, so as not to always favor the more established plugins. Perhaps if 20 ratings have been reached, then the number of ratings should no longer have any impact. We can probably safely assume that the rating has evened out after 20 ratings. For example, if a 4.5 star plugin has 200 ratings, then it shouldn't have much of an advantage over a 4.4 star plugin that has 20 ratings.

Change History (13)

Do you have any examples of searches and poor results where you think this will make a significant difference?

Otherwise I think this will just encourage plugins to ask for more ratings.

It came up earlier to just look at total number of one star ratings rather than trying to boost. Or on whether the ratings got replied to. Whether something is getting replied to seems like a good metric really. It indicates that the plugin author is paying attention, and is trying to give real support to real users. It is also something that is under the plugin author's control and doesn't reward folks for nudging lots of users to just give them reviews with no content in them.

I think this mostly happens in some-what long tail searches where there are fewer big players. The advantage also isn't enormous, but it is present.

My suggestion isn't so that plugin creators would resort to ask for more ratings, I'm not in favor of that. But instead, it's so that a having a very few number of ratings - maybe by the author themselves, maybe a friend or a handful of testers or early adopters - won't result right away in an advantage. That's why I mentioned a cut off like 20 ratings (I just pulled that number from the top of my head). My logic is that if a plugin has lower than 20 ratings, the overall rating should have a lower impact than others with 20 or more ratings. But it doesn't mean that having 300 ratings should always trump a plugin that has 50 ratings. After 20 ratings, then we can assume that the overall rating has normalized to a stable value.

For example:

(I've got nothing against these plugins, I'm just using them as examples)

Awesome Responsive Menu is higher than WP Mobile Menu. It may have something to do with the name of the plugin not matching the word "Responsive" though. But I think the amount of ratings (6 vs 45) should make WP Mobile Menu appear higher.

Eazy Under Construction with one 5-star rating outranks a lot of similar ones in page 2 who have 30+ ratings. Although the naming may have something to do with this, maybe the rating bumped it higher.

As for replying to ratings, I'm not sure about this. I imagine creators replying generic stuff like "Thanks for the rating, we'll do our best to blah blah blah" to 5-star ratings and actual replies would go to 4-star ratings and lower. Replying to support threads might be a better way to go, although this is taken into account already.

Great examples, thanks. I'll take a look at these in the context of other changes. It's all pretty small adjustments which I think are going to be really hard to hand tune. But I'd bet that we have a lot of cases where some minor adjustments would help.

Note: we encourage plugin authors to "ask" for reviews. We only tell them not to pay for them in some way, even by offering free stuff in exchange. We actively remove reviews that are paid for when we find out about them.

@gibrown I think it's pretty much about the small adjustments and refinements now since the majority of the search query put out great results in the most part. I would put out a patch but I cannot find anywhere the indices being used, I found a gist that you posted 8-something months ago, but that seems outdated already. I think those are private and in wpcom.

@Otto42 Let me correct myself, what you said is what I meant. I've only added the reasoning that a small amount of reviews won't be a good enough sample size or indicator of an overall rating :)

If you have adjustments to the query you can also just try post them on here and then I'll try them against the 3000 queries I've been running against during my testing to see how they work on a wider scale.

The 20 there is an arbitrary value and can be replaced with anything which can be deemed as a minimum sample size for an overall rating, it can be 10 or 8. This is to prevent plugins with only one or a handful ratings from being favored. I can't find the field for the number of ratings, so I placed it as ratings_num above to illustrate what I have in mind.

At a quick glance, my main concern is that having a fixed number as a filter creates a step function and I think that in general we'd do better to have smooth functions in how we rank things. So I'd rather have a decay for small numbers of ratings.

Good point and thanks for the links. Here's a smoother one that adds a multiplier 0.75 to 1.0 when it's below 20. This one actually works with the script you gave. The values for the range filter 20 and the offset and decay can be further fine tuned. The decay can be increased to a higher value to lessen the impact of this filter.

I think I will try putting something like this into the index. I need to figure how to make it work on a 5 star scale, but this addresses a lot of my concerns with hard coding any limits. Noting here mostly as a reminder.