Peer Review as a Reputation System

Many people reacted to my take-down of peer review, and at least one commenter asked, "Can you come up with something better?"

Easily. The generic problem is something called "reputation systems."

For example, Google's page-rank formula is a reputation system. If a particular page has the keywords you are looking for and a lot of links, it shows up at the top of Google's search results,.

E-bay has a reputation system. Participants give ratings to transactions, and you can see the average rating of someone with whom you are about to transact business.

Peer review is a reputation system. It is closely tied to another reputation system, called tenure. Which is closely tied to another reputation system, called academic hiring and promotion.

I have issues with all of those reputation systems. Keep in mind that this could all just be sour grapes on my part.

I think that academic hiring and promotion create too much concentrated power among just a few grad schools and a few people at those schools. Part of the reason that macro is so desolate right now is that you can trace just about every prominent macroeconomist today to about half a dozen professors who were prominent at MIT, Chicago, and Minnesota between 1975 and 1985. That would be fine if everyone were mining an area that is rich in gold. But they were all mining an area that was devoid of gold.

I think that the focus on publications for tenure does not promote wisdom. A lot of tenured professors are clever without being wise. In fact, there is no way for wisdom to enter into the tenure equation. But if I look back on my own education, I place a very high value on the few professors who had wisdom and very low value on a lot of professors who had nothing but cleverness.

All that said, I think that the average outcome of hiring and tenure decisions is quite good. Lots of individual injustices, but the overall result tends to be reasonable.

However, of all the reputation systems at work in academia, I think that peer review for journals is the worst. With all of the modern tools available for disseminating ideas, commenting on ideas, rating ideas, citing ideas, and doing tabulations of all the ratings and citations, it is easy to come up with better systems.

For example, suppose I do some empirical research and report the results in a paper on line. Other people can rate my paper in terms of reliability, importance, or other criteria. Those ratings can be much more reliable than peer review. The system would require incentives for people to be honest. It probably would require a reviewer rating system so that the reviews of more reliable critics are weighted more highly than the reviews of less reliable critics.

If academics were a free market economy, I would expect peer-reviewed journal articles to die quickly and be replaced by alternative reputation systems. But the current system advantages incumbents enormously, and there is no reason for them to give up that advantage.

The hard part is not coming up with a better reputation system than peer-reviewed journals. The hard part is getting adoption of a system when the people who would have to adopt it have no incentive to improve the system and every incentive to leave it as it is.

The amount of academic work that is not related to academia or peer-reviewed journals is very large. This may be more prevalent in my field (comp sci), but one huge advantage of electronic forms of publishing is clear to me: instant feedback. Many engineering proposals are published as request for comments, or RFCs. However, in engineering the incentives seem to be aligned much more closely with actual results, instead of pursuing reputation as an end unto itself.

I did a lot of thinking about how to build a reputation system for academic ideas before I realized a ton of people smarter than me were doing the same. There seems to be a lot of working going into trust and reputation metrics, as there are many applications for this technology on the Internet. I would think this is an area where economics (or maybe just economic thinking and game theory) could contribute.

The hard part is not coming up with a better reputation system than peer-reviewed journals. The hard part is getting adoption of a system when the people who would have to adopt it have no incentive to improve the system and every incentive to leave it as it is.

I think you just have to let new people adopt the new system and the older people become irrelevant.

Burger flipper,
AP econ is painful to teach. It's a fossilized curriculum from the 1970's. I don't teach it any more. AP stats is fun to teach. I enjoy my high school students. If I had college students like them, I would enjoy college teaching.

Grant, your last line reminds me of the comment attributed to Max Planck to the effect that science advances funeral by funeral.

Burgerflipper,
I am one of Dr. Klings AP students who went on to major in econ.
Granted, I took AP stat with him (which was quite enjoyable) and not economics. I had regretted NOT taking economics in high school up until today, when I read his comment about the curriculum.
Unfortunately, Bosh's No-Child-Left-Behind Act already had a model upon which they could determine their potential for success. The AP for most subjects just becomes a course where you learn how to take the test, without actually integrating the underlying material. As Dr. Kling said, the course hasn't been updated in decades. And won't be. Not as long as it remains something that students can profit quickly by through easy college "credit."

I am, additionally, curious to see what Dr. Kling and Caplan have to say about a college course "credit" system instead of an overall/specific "aptitude" test.

shashdot uses a reputation system, as do many other forums and blogging communities. There is a lot of work going into making better reputation systems.

One thing I've been thinking about is how a successful academic reputation system might replace journals, if government-subsidized academics still had to publish in journals. I suppose what would happen is that much of academia would slowly become irrelevant, but it wouldn't ever go away because of state funding. Am I the only one who thinks this has already started to happen?

Blogging software: Powered by Movable Type 4.2.1.
Pictures courtesy of the authors.
All opinions expressed on EconLog reflect those of the author or individual commenters, and do
not necessarily represent the views or positions of the Library of
Economics and Liberty (Econlib) website or its owner, Liberty Fund,
Inc.

The cuneiform inscription in the Liberty Fund logo is the
earliest-known written appearance of the word
"freedom" (amagi), or "liberty." It
is taken from a clay document written about 2300 B.C. in the Sumerian city-state of Lagash.