Disaggregating
Quality Judgements

Something
has quality if it might be what
you want, but this can never be the whole story unless your needs are identical
with those of others. Quality is a rough proxy for an item’s suitability with
respect to your needs, based on the fact that your needs will have a lot in
common with those of others. If everybody’s needs are different, for example
needing the right word to express something, then quality has no meaning –
there is no such thing as a good (or bad) quality word. If there is a
substantial commonality between the needs of different people, for example that
an apple is ripe and not gone bad, then an object that fulfils what is common
to those people’s needs can be judged as to its quality – thus one can have bad
apples. I will call selecting objects as to their quality the quality heuristic.

The
quality heuristic goes like this. A person, persons or process make judgements
on a set of items as to their quality. All those searching among these items
for something that meets their needs can reduce their search cost by
concentrating on those judged as having higher quality. A version of this is
illustrated in Figure 1.

The
quality heuristic becomes important when it is costly or impossible to find an
object that meets your needs yourself. For example, if only a few items can be
made available for many to consider then it is important that those items that
are selected meet the needs of as many of the potential users as possible.
This was the case when publishing an academic paper and distributing it was
costly – only a few papers could be published so it was important that only
quality papers were. Such quality judgements were possible because there was a
lot in common between the academic needs of the readers – they all wanted
similar things, such as: soundness, importance, novelty, verifiability etc. Thus
bottlenecks to finding and accessing what one needs makes the quality heuristic
necessary.

Of
course, there may be only a partial alignment between the needs of potential
users. So in the apple example, although almost nobody wants an apple that is
not ripe or has gone bad, some will want an apple that goes well with their
cheese, some a small sweet apple for a child’s lunchbox, and some an apple for
making an apple sauce. The more closely that a quality judgement is restricted
in context so that the needs of users in that context coincide, then the more
useful (and hence more meaningful) a quality judgement can be. Thus it makes
more sense to talk about an excellent cooking apple than simply an excellent
apple (unless the context of use has been implicitly or explicitly specified).
The greater the commonality of needs, the more effective the quality heuristic
is. For a system that produces a binary or single dimension of quality
judgement then the commonality between the users is critical to its success, it
works depending on whether what is suggested by one member of the community is
actually what another member wants. However when one disaggregates the quality
judgements this becomes much less of a problem.

With
respect to academic information, there is no longer a great cost to the
distribution of papers (Harnad 1998). However, users still have a difficulty
in finding the information one wants due to the sheer quantity of papers
available – it the search cost that is the bottleneck now. Thus there is still
a need for quality information, but due to the different bottleneck, there are
better solutions that the one illustrated in Figure 1. The users still need access to the information from quality judgements, but there is no
need to impose a filter on the distribution of information. Rather if rich
quality judgement on the items can be associated with them and made available
to users (say in a database of judgements plus link to the item) then the users
can use this information in a flexible way to find those items that meet their
needs. In this way the needs of users can be met in a more accurate manner,
dissolving binary quality judgements into a richer and more informative set of
judgements with respect to different aspects of the items. Thus if judgments
as to a paper’s novelty, importance, completeness, clarity etc. were captured,
then one user could search for all items that were very novel but with only a
minimum level of clarity whilst another could insist on high completeness but
not need any novelty. Monolithic quality judgements that averaged all the potential readers needs would be
replaced by a more flexible system, but one where quality judgements are still
central.

Figure 2. Delaying selection using
rich quality information

In
this way items are tagged with extra, judgmental information as to their
qualities,
that allow the users to solve the search problem, but in a way more specific to
their needs. The binary publish/don’t decision is replaced by that of making
quality judgements along many dimensions. This need not be done my one person
or institution, and could be decentralised. Indeed the user could select which
kinds of sources of judgemental
information they will use. Thus social simulators might choose to only utilise
judgements by a specially formed panel of reviewers, from any other social
simulators, or indeed from anyone at all. There is nothing to stop the
providers of judgemental information being judged in a similar way to the items
that are judged. One need only have to trust a source of judgements as to the
trustworthiness of reviewers and that could allow one to limit the sources of
judgemental information used.

To
make such a system effective: there needs to be a protocol for encoding the
quality judgements in the various dimensions; a sufficient amount of quality
judgements need to be captured and stored in an accessible manner; the search
engines need to be able to utilise this information so as to allow for
quality-enhanced searches. The process of bootstrapping trust in the system is
essential to its uptake by users, the amount and soundness of the quality
information necessary to make the system at all useful. A feature built into a
browser could allow a pre-registered reviewer to simply set sliders as
appropriate to the paper they are viewing in the browser, optionally enter a
comment, and press submit. One could combine the output of Google with the
judgemental information to access the results, optionally browsing attached
comments of the results, before deciding to whether to read the paper. Google
has introduced a facility that allows the public annotation of any web page,
called “Google SideWiki” (http://www.google.com/sidewiki).
Given an agreed structure this could be in a form that could be (1) written as
above using review feature and (2) combined with search results, simply
removing those items that positively failed on the given quality criteria. I
first suggested this idea in (Edmonds 2000).

It
should be noted that the efficient selection and provision of papers to readers
is only part of the purpose of a traditional, distribution filter it also has
social aspects. A filter on papers may help the coherence and focus of a
field, by ensuring all its members read a similar body of work, it may be a
filter against dissent and criticism of a field, and it may act in political
ways allowing a collective determination of future directions and issues.

Edmonds, B. (2000). A Proposal for
the Establishment of Review Boards - a flexible approach to the selection of
academic knowledge. Journal of Electronic
Publishing, 5(4). http://www.press.umich.edu/jep/05-04/edmonds.html