@Omar In a whiney voice: But there is no badge for taking time. Seriously, though, it is likely that these bad edits are done by people who have no self-awareness. I.e. they are quite sure that their error is not an error.
–
LouisApr 25 '14 at 16:42

3

@Louis I know; but this will have a negative impact on innocent posters. Such reviews should be re-reviewed, however, this will be a hectic job as well IMHO. Btw, thanks, I just got a brozne badge! =D
–
OmarApr 25 '14 at 16:49

93

I like to be more direct about it: "When in doubt, don't review." Seriously, we don't need more reviewers. I don't know why Stack Exchange feels the need to incentivize/gamify the review process like this. I'm sure responsible people don't need carrots hanging in front of them for them to want to help review new contributions. I'd rather have 100 responsible users reviewing 1000 posts a day than 1000 badge whores racing each other over those same 1000 posts.
–
BoltClock♦Apr 25 '14 at 17:03

4

@boltclock I agree with you. Rep required to earn reviewing privilege should be revised.
–
OmarApr 25 '14 at 17:38

19

Actually, more seriously, perhaps we should make reviewing edited answers a higher (10k?) privilege. Editing answers is far more risky than editing questions, after all.
–
JoeApr 25 '14 at 17:48

3

What we need is a way to submit reviews like this to the honeypot.
–
Ben VoigtApr 25 '14 at 18:59

39

Sigh, somebody found a bad review again. Paying no attention to the thousands of reviews that went off without a hitch. This urge to get to a 0.0000% failure rate is inevitably going to be disappointing. And both jeopardizes and belittles the effort. Set a realistic goal, bring it up when it isn't met.
–
Hans PassantApr 25 '14 at 19:13

2

@HansPassant this OP is to encourage reviewers to use "Skip" I am aware as well as everyone here that there are many bad reviews goign on. Having a 0% failure rate is impossible, but reducing/decreasing such behaviour is possible.
–
OmarApr 25 '14 at 19:43

For the record, it was this kind of "racing for credit" thing that killed experts-exchange and doomed it to the subscription-based model that has been its downfall.
–
Dave KayeApr 25 '14 at 20:02

4

@HansPassant I routinely run through the Suggested Edits queue and find terrible edits that get approved by the time I get through the rejection dialog, and in almost all cases the reviewers who votes to approve have rep between 3k and 6k.
–
chrylisApr 26 '14 at 0:35

3

@MarcoAcierno: that doesn't fix the root of the problem, though. The bad editor still gets their 2 rep, the bad reviewers still get their badge progress, and their behavior is thus reinforced.
–
WoobleApr 27 '14 at 16:08

9

@BoltClock: the reason is that before we put these (fairly mild) incentives in, there was a massive backlog in every queue. What good is a queue with 100% accuracy if 90% of the tasks aren't getting done? True, there's less maintenance work needed for a system that no one uses, and I suppose it's a bit of an ego-boost for the handful of hard-core users who do step in... But useless as far as getting anything done. You can see this same principle at work in the close queue right now: with fewer reviewers, it's a struggle to even break even - just a few % more would turn that around.
–
Shog9♦Apr 28 '14 at 23:34

16 Answers
16

Martijn pointed the problem out pretty well in the comments. From my experience, the main reason why many people are reviewing edit suggestions is to simply get points for doing so.

That’s why, whenever I review stuff, I barely get a vote because by the time at which I decided whether or not I can actually make a clear decision on the change (which usually means looking at the question in detail, looking at some answers if available, and especially checking the comments), others already voted it through without thinking much. And very often, I would have clicked the reject button after my evaluation when seeing that it was already accepted by others.

A lot people are only taking part in this because they get points for actually voting, and what’s easier than just glancing at something and thinking “yeah, the editor did something that looks good” to approve it.

I’m not saying that people should be awarded to skip if they can’t make a good decision (because that would just result in abuse), but there should be more incentive to actually consider whether or not something was really a good edit or not. I don’t think the current reward to participate can do that though.

This question can be easily applied to other review queues, but I would just like to note that the "fastest reviewers in the west" problems are mostly in the suggested edit reviews. The close vote queue thankfully has a lot of time between votes.
–
gunr2171Apr 25 '14 at 18:41

1

I agree this is a problem - frankly, I don't think working the queues should earn reputation. If it's a really bad edit (as opposed to a pointless one) it may be worth reverting it yourself.
–
Chris StrattonApr 25 '14 at 18:54

1

@ChrisStratton I absolutely do that when it’s appropriate. But it would be better if such suggestions didn’t get through in the first place, signalizing all involved parties (the editor and the approving suggesters) that such an edit is not desired.
–
pokeApr 25 '14 at 18:56

I disagree somewhat with this post - I think at least part of the problem is that new reviewers aren't sure what the guidelines are. See my answer.
–
KevinApr 25 '14 at 19:17

4

Never knew working the review queues gains rep... I thought the most I could earn there was a silver badge in a year or two and a gold one in a decade... (Not that either is of any use)
–
DeduplicatorApr 26 '14 at 2:03

I think your words "A lot people are only taking part in this because they get points for actually voting" are significant. Perhaps there should be a time delay (say 45 seconds) between displaying for review and accepting a response. I often find my considered views on a bad post are ignored because it has already been approved by people who respond quickly.
–
AdrianHHHApr 26 '14 at 7:30

17

I started out trying to contribute by doing reviews, but I kept running into the same issue of the review being gone by the time I had completed my evaluation. I don't even click on the Review thing anymore - it seems to a waste of my time now due to the quick "approver" clickers.
–
GjeltemaApr 26 '14 at 16:30

4

@AdrianHHH I was thinking along similar lines. Assuming that a proper review should take some time, we should not punish people who actually do so. One way to achieve this is to pace reviews by releasing them slowly from the queue to the actual reviewers. If the initial reviewer(s) don't pick up the challenge, or their allotted time (as you suggest, 30-60s) expires, release it for other reviewers to pull.
–
Henk LangeveldApr 26 '14 at 16:48

@poke you just pointed out all that's wrong with awarding points for activity, then judging people by how many points they have (I've had snide IMs from people claiming I'm less than them simply because I've fewer points here...).
–
jwentingApr 28 '14 at 7:23

@jwenting Where am I judging people by their points?! I’m absolutely not doing that; I couldn’t care less how many rep others have. When I’m interacting with people (comments, votes, in chat, whatever), I only care about content, not their profile.
–
pokeApr 28 '14 at 11:14

"...That’s why, whenever I review stuff, I barely get a vote because by the time at which I decided whether or not ..." Glad that a high rep user feels the same as me. Actually, I was searching on meta if it is good practice to skip reviews if unsure. THX alot guys for making me feeling less unsure when clicking skip ;-)!
–
pc-shooterAug 31 '14 at 16:27

Poke's answer blames badge-whoring and gamification for thoughtless reviews, but there's another major incentive right there in his post, which he complains about but somehow doesn't recognise as a driving force behind the behaviour of the bad reviewers:

whenever I review stuff, I barely get a vote because ... others already voted it through without thinking much

I share his frustration at this experience, but I get around this (like Omar clearly does) by keeping an eye on all the suggested edits I'm qualified to pass judgement on (about half of which I actually get to vote on in time) and going back to overrule the reviews and revert bad edits that get accepted. Or, less often, to manually reimplement good edits that get perversely rejected, giving credit to the original author in my edit message.

However, it's easy to understand reviewers - especially new reviewers - being really nervous about doing this. What we do when we go back and revert the edits approved by others amounts to deciding to throw away the decision of a democratic process because we think we know better than the collective wisdom of five of our peers.

Let's suppose you're a new, naive, non-Meta-reading reviewer who wants to participate fairly in the review process, without using your new edit powers to overrule others. You click through to your first few tricky edit reviews and think carefully for a few minutes about each of them. What feedback do you get?

You get punished over and over for being too slow. You don't get your vote in in time, so you're denied any voice in the decision at all. Sure, you're not getting your shiny badges, but you're also not helping the site; the time you spent reviewing goes completely to waste. What's more, perhaps you take a look at how long ago the other reviewers made their decisions and see that they voted several minutes before you.

The feedback the review system has given you is that if you want to help, you're going to need to reach your decision faster. We want these suggestions turned over quickly, the site seems to be telling you, and there's no point in even participating in this system if it's going to take you five minutes to review a few lines of code changes.

How do you go "back to overrule the reviews and revert bad edits that get accepted"? With a Stackoverflow reputation of about 2500 I can review edits but have no real facilities to overrule. I could edit the post to improve it, but I cannot flag the group of users that approved the edit.
–
AdrianHHHApr 26 '14 at 13:56

@AdrianHHH I just mean re-editing the original post, or using the rollback feature to undo the suggested edit completely.
–
Mark AmeryApr 26 '14 at 14:59

1

While I agree with what you’re saying, you make it sound like it’s the fault of those who carefully think about their decision instead of rushing something out to get it done fast. But it’s actually the opposite. We should reward those that take their time and prevent others from rushing through the queue without thinking about the edits. The problem is that suggested edits are reviewed incorrectly because people don’t consider them well enough.
–
pokeApr 26 '14 at 15:15

8

@poke, I assume you're responding to Mark, and I think the two of you are in violent agreement. Mark just provides an excellent analysis of the problem, you're looking for a solution that addresses it. We could start by granting first time reviewers more time for their review, say 180s, before that edit gets presented to the next reviewer. In this case, I think that a slower pace for reviews would improve quality, and presenting them serially, and not in parallel could accomplish that.
–
Henk LangeveldApr 26 '14 at 22:44

The simple solution to this is to increase the number of reviews required to complete a given task, which we've been doing (slowly) for a while now. In fact, the only queues that don't require this are the First Posts and Late Answers queues, mostly because we allow voting there and allowing multiple people to vote proved disastrous early on. Still do-able, but... Gonna require tooling that doesn't currently exist. And yes, I laugh in the face of anyone who thinks we should allow voting in any other queue - this is why.
–
Shog9♦Apr 28 '14 at 1:11

5

+1. My takeaway, which I complete agree with and would like to see: Permit the reviews to take more time! I have taken time to come to a different conclusion than others, and I wish it counted.. because I'm right!
–
maxwellbApr 29 '14 at 16:21

As a relatively new reviewer, I have a very different perspective. Quite simply, I'm not sure what the guidelines are for using or not using the Skip button are! I've intuited that I should use it whenever I'm not sure what to do, but frankly, skipping feels lazy. It feels like I should have known what to do or spent enough time to figure it out. Moreover, I don't really remember reading any FAQs, guides, help center articles, or meta posts explaining what the guidelines are for using the skip button or soldiering through to figure out how to handle a difficult review.

What I think this means is that a simple click-through or FAQ article saying that it's OK to use the Skip button when we don't know what to do with a post would take care of the problem. This will take a lot of pressure off of new reviewers and encourage them to skip more liberally.

Yeah, I think a big issue is also that reviewers don’t get a feedback on what they reviewed. It just silently disappears.
–
pokeApr 25 '14 at 19:51

13

Skipping isn't lazy. Doing reviews is a voluntary contribution, and you are always excused for not having the time or skill to perform a review. The only thing that counts as lazy is getting credit without having done the work, e.g. voting on something without reading it in order for your review to count for a badge. As long as you don't do that, you aren't lazy. @poke You can browse through the outcome of your reviews in the “activity” tab in your profile, subtab “reviews”.
–
GillesApr 25 '14 at 21:02

9

The guidance for reviews reads “Skip — if you are not sure and want to go to the next question”. How do you think it could be made clearer that skipping is perfectly ok?
–
GillesApr 25 '14 at 21:04

@Gilles You can do that, but honestly, who does that? Most will be happy enough with knowing that it gave +1 to the review track…
–
pokeApr 25 '14 at 21:42

2

I think I would feel a lot more comfortable with the policy - and so would other users - if there was a note about skipping in the help center. The description text for the review queue feels like reading an instruction manual for a machine – it's all the things SE is required to tell me, but it doesn't give me a good sense of how people actually use it. Again, a FAQ, help center article, or canonical meta post would clear that up pretty quickly.
–
KevinApr 26 '14 at 1:43

Perhaps this discussion could be the canonical meta post. As long as enough people are directed to it...
–
KevinApr 26 '14 at 1:44

No, there's no shame in using "Skip". I use it all the time. A very simple way to encourage people to click on "Skip" more often would be to simply make it a more prominent option by moving the button to the left. Currently, it's always the right-most button looking something like:

[Looks Good] [Edit] [Recommend Deletion] [Skip]

From a UX standpoint, this makes "Skip" the least desirable option. I'd consider to move "Skip" to the left, maybe after "Looks Good", maybe even to the left-most, "first-choice" position.

I have always used the "Skip" button when necessary. Sometimes I come across an Edit review and someone has fixed code in an answer that I don't understand. It's not helping anyone for me to approve it if I have no idea what the results of the edited code will be!

Recently there was a post about a guy who get banned for missing an audit. I think his comments were very telling when he said he'd been banned twice before for missing audits and he was "only 2 away from a silver badge". Too many people are gaming and not really caring about the reviews.

Maybe the audits need to be more subtle. The current Edit audits are blatantly absurd.
–
faddenApr 26 '14 at 1:19

They're absurd because either you're reviewing and thinking about it or you're not.
–
JoeApr 26 '14 at 3:28

1

@Joe still, the current audit system could lead to green distribution detection rather than actual reading.
–
Jan DvorakApr 26 '14 at 3:58

When an edit change the code of someone => reject it. No matter if you know what is it, or what he changed... if he wants to point out something wrong in the code he should post a comment.
–
Marco AciernoApr 26 '14 at 17:59

2

@MarcoAcierno - Not necessarily. If someone posts an answer that no longer works due to program revisions, it should be updated. Otherwise the (unedited) answer is essentially wrong.
–
Johnny BonesApr 28 '14 at 16:55

1

@JohnnyBones Personally i found better to post another answer with the new code. (if by someone else, but the author could choose to update it) but the for me the important thing that is the old code is still there.
–
Marco AciernoApr 28 '14 at 16:59

@MarcoAcierno it's not uncommon for me to replace verbose code with an SSCCE, and I stand by that.
–
Mooing DuckApr 29 '14 at 16:10

Agree with @fadden - you barely need to be awake to notice the edit audits because they're rarely (if ever) anything but blatant random word spew. I think more subtle edit audits (that require some level of thinking) would help weed out more robo-reviewers.
–
Andrew MedicoMay 8 '14 at 20:07

I sometimes go through my whole review queue hitting skip :-) Most edits require me to understand the subject matter, and I sometimes wonder why StackOverflow is showing me some of these. (I assumed it was a lack of reviewers, or desire for multiple opinions, but other comments and answers seem to say that is not it.)

It would be nicer if it only showed me questions where I have asked or answered questions on all of the tags there. E.g. I don't really do Java any more, but [java] [sentiment-analysis] I can usually intelligently evaluate. But [java] [some-library-unknown-to-me] is an immediate skip from me :-)

Here here! I stay almost entirely in [C++], but then I get reviews for [ML] and I think "is that a programming language? I've never even heard of it!"
–
Mooing DuckApr 29 '14 at 16:11

2

+1 Absolutely. There is no way I'm going to make a decision on a topic I barely understand unless the edit is obviously nonsense. I just skip, skip, skip through to the ones I feel qualified to comment on.
–
DNAMay 3 '14 at 21:32

1

+1. Perhaps tags used in less than X questions should be excluded, if there is other tag in which the user answered something (and got some reputation). This should be voluntary on betas and other smaller site (otherwise there would be no one to review most edits), but it is befinitely good for SO. Or it could be voluntary on SO too, but only tasks for tags where user has some reputation could count for badges.
–
Pavel V.May 7 '14 at 12:23

I agree with many of the answers given, but would also like to answer the question at the top of the post:

Is there a way to encourage reviewers - especially new ones - to use "Skip" button when they are hesitant and not 100% sure what to do?

(My boldface.)

Yes!

Rename the button to "Not Sure".

While the current button texts

Looks Good

Edit

Recommend Deletion

Skip

represent actions taken by the reviewer, "Skip" is actually something of an odd duck here: it specifically represents the action of "do nothing with this X, not even saying that it does not need any action".

The reason why someone should pick that option is because they aren't sure what to do with the post in question.

Making the button text always read "not sure" rather than "skip" makes it clear when it should be used. That alone might be enough to encourage its use by reviewers.

Preface:

Since this a discussion, I would like to share my thoughts over reviewing in general. I am not new to this community nor am I old; I am learning new things every single day. I spend several hours on SO mainly to answer questions and follow-up on my answers, I also check review queue when things get calm.

There is no doubt that "Stack Overflow is moderated by you" reflects professionalism of SE and its' staff behind the scenes; additionally, it is a brilliant idea and has definitely contributed to making SE a nucleus for persons seeking out help and advice as well as keeping comfortable and organized place to share expertise. Moreover, it has implanted "loyalty" in its' frequent users.

However, at the same time, it made some users race to collect reputation to serve other purposes rather than effectively utilizing their privilege in helping the community remain clean.

I always wanted to say what I am going to say not, although, I assume it wouldn't be accept by the majority and they would think that I would like to change the rules. I better let it out instead of keeping inside.

It is relatively easy to gain 2,000 rep, which entitles one to review edit queue. In fast-paced and high traffic tags, it is possible to reach 2,000 in a short period, 7 to 10 days maybe? But, does that mean or indicate that such a user has gained enough experience to distinguish between good or bad edit? This also applies on flags handling and other types of reviews.

In light of the above, I would like to brainstorm the following points:

Create a "Training Center" where users can practice reviewing and flagging, rather than practicing on real posts.

To consider other criteria / factors in addition to reputation for entitling users to review. E.g. number of visited day or consecutive.

In case of "edits" final approval should be taken by post owner and/or high rep user (10,000+)?

Make real use of badges. For example, users with golden badge of a specific tag, has extra privileges over other users in that "tag"?

Whether you see those points feasible, do-able and may reduce such accidents, I would love to hear your comments.

Well a systematic problem should have a systematic solution (or a set of solutions)

One possible (partial) solution to decrease number of incorrect approvals could be not to display entries "inappropriate" for a particular user. Suppose if I don't know anything about PHP or Wordpress (and have them in ignored list), likely I won't make a reasonable, well grounded estimate if a particular edit is correct and really improves the question or the answer.

Certainly most of edits I've seen so far are spelling corrections and formatting improvements. Probably they should be treated specifically, regardless of their "payload" and the set of tags assigned to the topic. Perhaps such edits should be marked specifically, so only people really interested in linguistic beauty could spend their time on such edits.

BTW even being a non-native English speaker I can't miss the fact that a significant number of spelling corrections are, well, far enough from being State of the Art of English language ;-) But most of the topics are pretty understandable so I guess the clarity and beauty of the English language isn't that important on a technical international community-driven site (my apologies to those who fill pain reading text with spelling mistakes or simply "Bad English")

As poke pointed out: looking at the actual question, other answers and comments helps deciding on the vote. It may give you a perspective on why someone wants to make a given edit. Currently however the review page shows you only the changes. You have to click on the question to open it in another window to get the full context.

While being one-click away is not far, I believe the context should be given immediately on the review page (question, answers, comments), to give people incentive to even look at it!

Permit late reviews

An issue that many have raised seems to be that if you spend more time reviewing a single item, the item may already get closed for reviewing and your vote is simply ignored. I think it would not be too difficult to implement a mechanism for late reviewing though. It could work more-or-less like this:

If you open an item on a review, you are (almost) guaranteed that your vote will count.

When enough votes are collected, and the change passes, it is applied.

But, if your late-vote is against the majority, the change may be automatically reverted and the voting is reopened until enough abs(positive-negative) votes is reached again.

The late-review vote would not be accepted only if there were further changes to the item, stacked upon already accepted change.

This approach would not only give credit to those that do spend time on reviewing each item, but it would also encourage robo-reviewers and those who want to collect points to actually spend some time as well since they are going to get the points anyway - there is no race.

Secondly, a form of late-reviewing already exists in hands of those who do reediting or rolling-back a change thanks to their high rep anyway.

Save for later

In the view of late-reviewing, an additional button could be given "save for later", stating that you want to review an item, but you need a bit more time for it, and perhaps you want to do it later. It would work similarly to "skip" but would keep it attached in your account allowing you to return to it easily later, and with a voting option open.

Draw attention of experts to disputable changes

Perhaps, if an item gets mixed reviews it should be somehow advertised to experts in the fields (high rep in given tag for example?)

Suspicious/Unsure

Perhaps, a "suspicious"/"unsure" button could also be introduced, indicating that you find a given edit "suspicious" and may need more attention from a bigger group of reviewers before it is accepted or rejected. This could also trigger the "draw attention of experts" behavior.

@Omar skip-250 is only a trick for robo-reviewers (no badge requirements can't help against them, it's a job for review audits). For sensible reviewers, this would merely offer a chance to pick a good habit. I had similar experience with Electorate badge, which requires one to vote certain amount of questions: at first I robo-voted questions indeed (back then, I knew only how to vote answers) - but soon this became boring and I learned to evaluate questions, and I discovered that this is the right and fun thing to do, much better than robo-clicking
–
gnatMay 22 '14 at 15:50

Just because there's no shame does not mean there are no negative consequences. If you skip something, you will never see it in the queue again. So if you want to let it sit in the back of your mind for a while and come back to it the next time you're in the queue (if it hasn't been solved yet) which while not exactly a problem on SO is a concern on other sites, you can't do that. You either have to try leaving and refreshing the queue and hoping you don't come back to it right away, or you have to sit there until you are certain what you should do with it, or you will never see it again.

So I would propose getting rid of the skip button entirely. I would replace it with a "Come back to this later" button. Then it won't show up in your queue for at least another 10-15 minutes and/or 3-4 reviews and you can approach it with a fresh set of eyes.

The skip button should be used when you don't know enough of the technical details. For example, when an edit introduced a new tag and you have no idea what the tag means.

But there should be no need to use the skip button for the example posted, because the edit attempted to change the contents of code. All edits that does this should get rejected (invalid edit or radical change), with no exceptions.

First of all, changing the meaning of other people's posts is considered rude in most cases.

If the original code in a question contains obvious errors, you should not fix it, because the errors in the original code might be the very source of the problem. Or they might indicate that the poster has not even tried to compile the code. Instead, leave a comment.

If the original code in an answer contains errors, you should not alter the code, but instead post a comment: "didn't you mean this instead?". Perhaps the answerer had a reason for doing what they did. Perhaps they made a mistake, but they know a better way of fixing it than you do. Leave the decision of changing to them.

It is not ok to edit code to suit your personal taste in coding style. Coding style is very subjective and there is no obvious right or wrong. Also, a poor coding style might be the very cause of the problem, as in the already famous Apple bug.

It is however fine to edit code for fixing non-existent indention or formatting, as long as you do not change the actual meaning of the code.

Unfortunately, correct vs. incorrect isn’t always that simple. Even after careful evaluation, there often isn’t one right answer. And punishing reviewers for having a different opinion shouldn’t happen. We should only try to get rid of the careless reviewers.
–
pokeApr 26 '14 at 17:25

Indeed, but the minus rep would, I'd hope, put a huge break on robo-reviewing. If genuine reviewers get a minus rep every here and there, I'd imagine they won't care — I certainly wouldn't.
–
Denis de BernardyApr 26 '14 at 19:33

4

-1; you're tipping the queue heavily in favour of rejection while punishing non-conformity in an attempt to punish inaccuracy. Now, conformity might be a good metric of accuracy if most reviewers make the right decisions, but if you set up a system that ensures that approval is both more difficult to achieve and always incurs a risk of punishment, but rejection is easy and safe, then it seems obvious to me that the outcome is going to be honest editors getting punished over and over for trying to approve edits that the robo-reviewers reject.
–
Mark AmeryApr 27 '14 at 9:14

1

There should be better solutions than "punishment".
–
OmarApr 27 '14 at 13:20

@Omar: Presumably — would you mind suggesting one instead of simply broadcasting that robo-reviewers should use the Skip button? I'm sure there are better ways to make sure a reviewer is paying attention than to show invalid edits, too. And yet, here we are. :-|
–
Denis de BernardyApr 27 '14 at 16:40

1

I have added some points in my answer. However, I found out that who accepted the edit are members since 4 yrs but have low rep, which leads to another problem that also old members are not paying attention and I assume they don't care about badges.
–
OmarApr 27 '14 at 16:48

Perhaps not vote, but a way to flag the actual edit. I've often flagged posts where one of the editors approved crap edits and they have a 100% approval rate. But it isn't the actual post that should be flagged, but rather the editor/edit. In many cases the edit review system is too crude to weed out the robot-reviewers, so they might need a manual edit ban by a diamond mod.
–
LundinMay 6 '14 at 11:32