Nowadays, conferences are being run using a century-old approach that puts a lot of trust into the reviewers and the program committee.

However, examples of successful social networks like Slashdot, Digg and Reddit show that it is possible to implement an system that trusts none but works well enough to push up the most insightful texts.

The modification of the evaluation processes shall evolve over at least
three stages.

First, an existing system shall introduce the ability to review the
reviewers, that is, vote for the quality of reviews and thus -- for the
quality of the work done by a particular reviewer. This will allow to
rebuild trust in the reviewers in a transparent way.

Once the reviewers are less of a problem, the major attack should be
done on the selection itself. The only reason that only a handful of
submissions are published among the others is the limited amount of
space in the printed journal. It is time to allow everyone to publish
their papers on the conference website, ranked by the reviews, so that
the selection is gradual, instead of binary. Until paper publications
naturally die out, a cutoff ratio or number can be kept.

The last and the most difficult stage is to abandon the reviews by
dedicated reviewers all together and move onto the review by peers
where everyone who publishes can review peers in the same
conferences openly. A fair share of meta-reviewing is expected for this
system to keep up against abuse.

Here is a fictitious story of a business client that orders software for a mobile phone to an embedded developer.

Take such a trivial feature of a mobile phone as a the possibility to store last calls. What do you know about it besides that it allows to view last incoming and outgoing calls? This is the level of knowledge that a typical client has and he will express it as a requirement the same way:

— I want that my mobile phone software to store last incoming and outgoing calls.

The developer has two choices: either assault the client with question based on unconfirmed assumptions or tacitly do exactly what was asked. He writes code that keeps two lists ordered by the call time, one list for incoming calls, another one — for outgoing calls.

Soon after, the client finds out that a competing company uses a different design and demands the software to display:

all calls together with their destination (incoming or outgoing)

missed calls

received calls

outgoing calls

rejected calls

Unfortunately, the previous code has already been written and is linked tightly to the UI code and to the rest of the mobile phone's OS.

So, instead of scapping the previous design and starting from scratch, the developer is now bound to implement new data structures that consists of one new list that combines incoming and outgoing calls in the right order to satisfy the first additional requirement, and three more lists for each of the 2nd, 3rd and 5th additional requirements.

This seems like nothing, but coupled to the synchronization code and error handling code, the result is likely to look cryptic for everyone, including its own developer.

I tried ranking users by both average and median comment score, and average (with the high score thrown out) seemed the more accurate predictor of high quality. Median may be the more accurate predictor of low quality though.