Hi folks, if you haven’t encountered me before, I’m Justin French, Product Owner of the Marketplaces. As I’ve previously mentioned in other threads, the rating system is the next major pain point we’d like to focus on in improving the Marketplaces this year. There’s certainly no shortage of feedback and ideas!

Like many recent projects, we’re taking this potentially epic piece of work and splitting it into smaller pieces we can confidently ship really soon, and some more we will look at later.

Show a breakdown of the average

This is an incredibly popular and long-running request! We’re redesigning the sidebar of an item page to accomodate a little more detail, rather than just the overall average rating — buyers will see the number of people who have rated the item at each step (5 stars, 4 stars, etc), the total number of ratings received and the precise numerical value of the average (e.g. “4.6” or “3.9”). This work is mostly complete and will ship soon!

Rounding to half stars

Also requested, we’ll start rounding to the nearest half-star rather than whole star. Currently an average rating of “4.6” is rounded to “5”, “4.3” is rounded to “4”. By rounding to the nearest half star, an average “4.6” or “4.3” will be rounded to “4.5” instead. Some items will go “up” a half star, and others “down”, but overall, the graphical representation will be closer to the real numerical average. This work is mostly complete and will ship soon.

Encouraging more ratings

We want to encourage and remind buyers to provide a rating for any of their unrated purchases. We’ll start with an email follow-up a week after purchase and measure what effect this has on the number of ratings per item. This is also mostly complete and will ship soon.

What are we rating, exactly?

We don’t explicitly state anywhere what a buyer should be thinking about when they’re rating an item. Is it the technical quality? Aesthetics? The author’s support? Flexibility & configuration options? The author’s willingness to help with questions or customisation? The purchasing experience? The installation process? Value for money?

To help us better understand what’s already taking place, we’re going to run a survey, interview a range of authors and buyers, and start collecting the primary reason (selected from a short list of predefined options) from buyers as they rate each item. There’s no plans to display this data on the site (not even to the author), but it will help us better understand how the rating system is currently being used, and to plan our approach to some of the remaining feature requests.

Speaking of feature remaining requests…

Who rated this, and why?

An often-requested feature is buyers providing a comment when they rate an item, displayed alongside their username — some propose this should be required for all ratings, others just for one and five star ratings. There’s a lot of hidden features we might need here too (moderation & dispute resolution, marking reviews as unhelpful, etc).

The underlying goals of these requests are:

improve transparency for everyone

provide authors with insights to help improve their items

provide relevant ratings to buyers to help their purchasing decisions

These goals are important to us, but we’re not starting on the build right now — the data we’re collecting and the research we’re working on will help us design the details of a system that best meets the community’s needs, as well as ours.

We’ll update you with what we’ve learned (and our next steps) in a month or two.

Average averages?

We’re also working with a statistician to explore some alternatives to the existing method of averaging we use (including bayesian and median, which have been brought up a few times by the community). These are only experiments to review what happens to the data at this point. If it looks like we’re going to make a change in the future, we’ll keep you in the loop.

Ratings are forever?

Right now, a rating is forever — we calculate the average using every available rating since the item was first uploaded, which means it’s a mixture of historical and current ratings. We’re working with a statistician to explore the impact of ignoring older ratings. If an item has greatly improved over time, this might lead to a rating which is more reflective of what you’re buying today. Again, if it looks like we’re going to make a change in the future, we’ll discuss what we’ve learned.

Stay tuned!

I wanted to share this plan before we deploy the first batch of improvements so that you aren’t left wondering “is that it?”. There’s plenty more to be done, and this is the first step!

Edit: Made a small clarification to “What are we rating, exactly?” 4pm AEST (about 5hrs after the post).

Post Reply

<strong></strong> to make things bold
<em></em> to emphasize
<ul><li> or <ol><li> to make lists
<h3> or <h4> to make headings
<pre></pre> for code blocks
<code></code> for a few words of code
<a></a> for links
<img> to paste in an image (it'll need to be hosted somewhere else though)
<blockquote></blockquote> to quote somebody