1 Answer
1

What is the purpose of this test?

The test (known as a review audit) is designed to help new reviewers hone their moderation skills, while nudging more experienced users that don't seem to be paying close attention to what they're reviewing. Some people were in the habit of up-voting everything they reviewed without regard to quality; extremely low quality posts with a high score are very problematic in a system where the best information naturally rises to the top, so "audit" tasks are mixed in to make sure folks are paying attention.

How does the audit work?

First Posts, Late Answers and Low Quality - The test presents a reviewer with a post that was previously removed by a moderator and known to be of extremely poor quality (flagged as spam, not an answer or very low quality) or high quality (many up-votes, no down-votes, etc).

Close Votes and Reopen Votes - The test presents a reviewer with a question which the system has determined should be open or closed based on various criteria (highly-voted with no close votes, quickly closed with no answers, etc).

Suggested Edits - The test presents a reviewer with a randomly generated edit utilizing the Markov chain algorithm to randomly insert various words that may appear to be useful at first glance but actually make no sense and are meant to vandalize the post.

If a user fails a review audit, they are presented with a message that they didn't pass and encouraged to pay more attention in the future. If a user passes the test, nothing happens and they continue reviewing. In some review tasks, they are informed that they passed and thanked for their time and feedback.

The result is then recorded for analysis. Only diamond (♦) moderators or Stack Exchange employees can see the data; it is not accessible publicly even to high reputation users.

What happens if I fail?

For most, nothing. You'll be given some guidance on why the post was inappropriate and what you could have done instead, and then allowed to continue reviewing.

Repeated failures, especially in short succession can lead to your review privileges being suspended for a week or longer, depending on the frequency of the failures. Stack Exchange is continuing to analyze the data to determine the best action to take in these cases.

Do review audits count toward the Reviewer badge?

Passed audits count as a successful review toward the Reviewer badge. A failed audit does not count toward the badge, but does not decrement your progress; it simply adds none.

Do robo-reviewers dream of electric sheep?

Robo-reviewers don't sleep. They wait.

Important Note:

The information contained in this answer is subject to change as the review system is further refined. Changes that impact the mechanics of the test, or consequences of failing it repeatedly will be reflected in this answer as they're made, but not necessarily immediately after they're made.

I'd like a "I still disagree, your test if flawed" button for "failed" tests.
–
CodesInChaosNov 30 '12 at 15:33

12

@CodesInChaos: keep in mind, these test posts have already been deleted - so if you disagree strongly, you should probably flag them for moderator attention / bring them up on Meta / etc.
–
Shog9♦Nov 30 '12 at 17:33

24

I got the "failed" message when closing some questions. It presented a real one and I clicked on the "Close" button just to see how many other people had considered it a "closable" question, but it immediately said I failed the test despite not yet submitting a close vote... I quite often open that close dialog up just to see what other people are thinking (in the event I am overlooking why the question is being voted to be closed).
–
dreamlaxApr 24 '13 at 17:32

22

@dreamlax You should not review based on what others are thinking, because that only amplifies existing reviews. The review systems works because multiple people form an independent opinion. If there is (semi-)consensus that probably means the consensus is true and actions can be based on that consensus. However, if that "consensus" is because everyone agrees with the ones before him only the first independant opinion determines the consensus and thus it becomes worthless.
–
dtechApr 27 '13 at 12:15

16

@dtech: I see your point and fair enough. For me though, I don't blindly follow what everyone else is thinking, I only check to see if perhaps my mind has had a severe lapse of judgement.
–
dreamlaxApr 28 '13 at 3:02

14

Can users ever graduate out of having to take these tests? I've done hundreds of reviews, and have passed every test thrown at me. How long before the system figures out that I can recognize garbage? Maybe when I earn the Steward badge?
–
BarmarJul 3 '13 at 19:14

14

I suppose. It just feels a little condescending after a while.
–
BarmarJul 3 '13 at 19:22

25

I just failed my first audit by clicking "no action needed", on an answer that had already been deleted, which I knew by clicking through to the full question. Because my understanding of reviewing was to recommend future action (or to take action myself), I concluded that the problem had already been resolved, so no further action was needed. Have I misunderstood the purpose of reviewing, such that I shouldn't be looking at the live post to see the current state of affairs in its larger context? Is my misunderstanding of a different flavour? Advice/clarification would be appreciated.
–
nmjkJul 8 '13 at 19:52

7

Is it theoretically possible for the random algorithm to generate helpful edit suggestions? And, if that happens to me, can that be taken as conclusive proof that this is the matrix, and I am the Chosen One?
–
rootAug 15 '13 at 16:47

9

I was thinking about posting a new question, but hopefully a comment here will suffice. My problem with this is that I sometimes want to view some close reasons, which I can do before closing, except when these tests occur. I wanted to check if something was a duplicate by opening the close dialog, but was then "busted".
–
keyserAug 16 '13 at 14:11

3

Pffiew, happy to have read this. (I failed an audit because I was curious how much close votes an obvious good question got.)
–
NGLNNov 22 '13 at 21:50

9

@dtech Your point, that reviews should not be based on what others think, is correct. However, consider this case: I notice a question has had close votes, but see nothing wrong with it. Before I comment "this post is fine", I'd like to see why it was close-voted in the first place. For instance, it's considered a duplicate, and maybe it is? So I take a look at the linked post to double-check. It is only then that I can decide whether the post truly was a duplicate, or if people may have misjudged the post; maybe they only look similar, but actually ask 2 different things?
–
NolonarJan 29 '14 at 23:28

10

I would very much like an "I respectfully disagree" button added to the failure dialog. I suppose it may be a character flaw, but if someone solicits my opinion (as SO does by asking me to review things), and then tells me I'm wrong, I expect a chance to argue my position. It just seems polite.
–
MrTheWalrusApr 3 '14 at 23:25

3

I suspect that one real consequence of these audits is to encourage active members not to review. I decided to be a good citizen and do my share and clear off the review list whenever presented. But when you get one or two of these stupid and condescending responses a day, I decided whats the f***ing point and gave up reviewing. How many others have done likewise? You don't get anything for acting as a reviewer other than be insulted by the system for your contribution.
–
TerryEJul 12 '14 at 14:29

5

I don't get these at all; if the point isn't to go along with the herd, then what's the point of the test? If 5 people said a question is poor and should be closed, but I don't think it should be, then there's a disconnect. Either you are encouraging groupthink or you encourage people to make their own decisions, but the fact there's some hidden algorithm that determines if you reviewed "correctly" defeats the purpose of thinking independently.
–
Wayne MJul 18 '14 at 19:53