Share this story

Thanks to the Internet, we can rate, judge, and evaluate nearly anything we’d like—from products to movies to fitness centers—with minimal effort. In many ways, this is a boon, since there’s a wealth of information at our fingertips advising us what car to buy and whether or not we should tune into a new TV show. But behind each of these ratings is a person's opinion, and human nature makes those opinions notoriously fickle.

In the latest issue of Science, a group of researchers decided to see whether people's tendency to follow the crowd extended into online ratings. They tackled this question by carrying out a large-scale experiment in which they rigged the comment ratings on a website. The website they used is a news aggregating website (like reddit) that lets users rate comments and contributions by either “upvoting” or “downvoting” them. The comment’s score, which can be seen by all the website’s users, is simply the raw difference in the number of positive and negative ratings.

During the study, which lasted 5 months, the researchers manipulated the ratings that some comments received. Out of the 101,281 comments made during the span of the study, the researchers randomly assigned 4,049 comments (four percent of the total) to the positive treatment group, and these comments were given a single upvote upon submission. Another 1,942 comments (two percent of the total) were assigned to the negative treatment group and were given a single downvote; the rest of the comments made during this period served as controls. These proportions mimicked the natural frequency of up- and downvotes normally seen on the site.

Then the researchers left the comments alone and simply observed what happened. They monitored how likely the next user to see the comment was to give it an up- or downvote, as well as how the overall scores of comments in the two treatment groups compared to those in the control groups by the end of the study.

While the experimental setup was simple, the results weren’t completely straightforward; the effect of the manipulation depended on which treatment group the comment was randomly assigned to. In other words, a comment that was upvoted by the researchers fared differently than one they downvoted.

Comments that were randomly selected to get an upvote were much more likely to also be upvoted by subsequent users; the first user to see a positively manipulated comment was 32 percent more likely to upvote it themselves as well, compared to a comment in the control group. By the end of the 5-month study, the comments in the positive treatment group had ratings 25 percent higher than those in the control group. The researchers refer to this phenomenon as “positive herding behavior,” which essentially means that users tended to jump on the upvoting bandwagon, helping these comments accumulate positive ratings.

In the negative treatment group, the results were a bit different. Just as in the positively manipulated group, the comments that the researchers downvoted were also more likely to be downvoted by subsequent users. However, these downvotes were offset by what’s called a “correction effect”—the negatively manipulated comments were also more likely to receive positive ratings down the line.

Raters were nearly twice as likely to upvote a comment in the negative treatment group compared to the control group, suggesting that crowd correction can effectively neutralize one or a few negative ratings if they are undeserved. Thanks to this correction, the researchers didn’t see any “herding behavior” in the negative direction on downvoted comments, and the final score of these negatively manipulated comments didn’t significantly differ from the control group by the end of the study.

Obviously, comments that are particularly insightful, creative, or high-quality will certainly fare differently, as will those that are simply trolling. But thanks the large-scale, randomized design of this study, we can get an idea of the role that social bias plays without having to consider the quality of each post.

Overall, the researchers think there are three main phenomena that contribute to these findings. First, both experimental conditions increased turnout; in other words, a comment that is rated once is more likely to be rated again, no matter the type of vote it has received. Second, seeing an upvote does help users develop a positive opinion, especially if the user doesn’t have much prior experience with the comment’s author. Finally, users on the site had a natural tendency to upvote rather than downvote.

The researchers don’t specify which website they used, since its administrators have asked them not to. “They are concerned about the re-identification of individual users and the associated risk to their privacy,” the authors write. “Although there is a scientific need for the validation of results in these types of studies, there are competing concerns about the welfare of users.”

Ars, like many other sites, has instituted a comment rating system to encourage good discussion and has spent a considerable amount of time and energytweaking the system to make it work. And—in true Ars fashion—there has been livelydiscussion about the pros and cons of the system. The researchers behind this study hope that these findings will improve comment rating systems further, as well as help increase the sophistication of bias analysis in other types of social feedback, such as political polling, product reviews, and stock market prediction.

Share this story

Kate Shaw Yoshida
Kate is a science writer for Ars Technica. She recently earned a dual Ph.D. in Zoology and Ecology, Evolutionary Biology and Behavior from Michigan State University, studying the social behavior of wild spotted hyenas. Emailkate.shaw@arstechnica.com//Twitter@KateYoshida