Disclaimer: Any opinions expressed, potshots taken, or scientific views articulated are mine, and need not represent the opinions, potshots, or scientific views of the Federal Reserve Bank of St. Louis, or the Federal Reserve System.

Wednesday, January 18, 2012

Economics is an Open Science

Here's an article in the NYT which tracks a trend in the natural sciences and medicine (mainly) toward more open evaluation and discussion of scientific contributions. As Paul Krugman points out, economics has always been an open science, at least within the professional lifespans of people as old as Krugman and me. Even prior to rapid electronic communication, economic research was widely-circulated in working papers and at conferences and seminars prior to publication. The results were hashed over, reworked, and amended, with the ultimate published articles typically acknowledging the contributions of many other scholars on the road to completion. Publication was, and is, essentially a grade given to the work, i.e. a publication in the American Economic Review, Journal of Political Economy, or Econometrica, is A+, and you work down from there. Email communication and the internet served only to speed up the process and make work even more widely distributed. No more waiting for the Minneapolis Fed working papers in the mail.

As the NYT article tells you, things are not like that in the natural sciences or in medicine, for example, often because there are potentially large monetary gains and losses riding on the results. Natural scientists think that it is quite odd for anyone to be talking in public about their research results before they are published. Economic research can have very large implications for human welfare, but the path from a given research result to its effect on the public is typically so indirect and diffused, and the benefits so large to an economic researcher from claiming credit early for a given result, that no one is going to hide their work until it is published.

According to the NYT article, some scientists are complaining that the whole process of peer review and publication is "hidebound, expensive, and elitist." Do we have any of those problems in economics? I think "expensive" certainly applies, and the culprits in economics are the same ones as in the natural sciences. The NYT article states:

The largest journal publisher, Elsevier, whose products include The Lancet, Cell and the subscription-based online archive ScienceDirect, has drawn considerable criticism from open-access advocates and librarians, who are especially incensed by its support for the Research Works Act, introduced in Congress last month, which seeks to protect publishers’ rights by effectively restricting access to research papers and data.

Elsevier in fact owns many economics journals, and is well-known for extracting rents from libraries and subscribers. In the old print days, a publisher could provide useful services. Typesetting, printing, binding, and distributing print journals was a specialized task subject to increasing returns. Do publishers provide any value-added in 2011? Absolutely not. But as the article points out, publishers like Elsevier would like to pass laws like the Research Works Act that will allow them to continue to collect rents and impede the dissemination of scientific knowledge.

So, the expense of publication is a problem in economics. But is scholarly publication in economics "hidebound and elitist?" Krugman thinks so:

I have a somewhat jaundiced view of how the whole refereeing/publication system has ever worked; all too often, it seems to act as a way for entrenched doctrines to blockade new ideas, or at least to keep people with new ideas from getting tenure at a good school.

I'm going to disagree with Krugman here - not for the first time, of course. Editors, referees, and the economics profession can make mistakes. Sometimes it takes a while for us to catch on to a new idea and recognize what is important about it. Here's a good example. Expectations and the Neutrality of Money was rejected for publication by top journals (the American Economic Review, at least, as I remember the story), and published in the Journal of Economic Theory, at that time a relatively new journal with essentially no reputation. That article now has more than 3000 citations on Google scholar, was part of what Robert Lucas received a Nobel Prize in Economics for, and changed the methodology of macroeconomics in a rather dramatic way. The east coast economics establishment hated Lucas's paper (as Krugman still does) but that didn't stop the ultimate recognition of the work. For every inbred group of scholars eager to keep outsiders out, there are 5 journal editors who would be more than happy to publish new and pathbreaking work.

What about the blogosphere? Do blogs contribute to economic science? They certainly could, but most blogging on economics, including mine, is journalism. I'm very pleased that I don't have to acquire space in the Washington Post, the Wall Street Journal, or the New York Times to speak my mind, but I don't think that what I do here is serious science. Serious economic science is what is published in the top economics journals.

Peer reviews in academic publishing are also important for improving the clarity of the scholarship and making sure the research is ready/deserves a wider audience (or not). Sure the people at the cutting edge of a particular field should keep up with the latest working papers, but journals are a good place to read the literature.

Still I am hesitant to say where "serious science" is taking place...sure research can be a long slog that ends with the journal publication, but the best work starts with a clever idea and those ideas could come from a blog conversation.

Yes, I agree. There is a role for journals as a filter for people who are not going to all the conferences, or who want papers in polished form. I'm probably selling myself a bit short too. My hope is that my discussion of the issues in this blog might actually give someone an idea that they can use in their research.

The quantitative importance of having more "open access" journals is unclear to me. This seems quite different to me than open access to fiction writing or pop music.

Do we really think there is large proportion of people who (i) can read technical journal articles AND (ii) have no access through their institution to a "free" (in terms of the marginal cost they privately pay to view an article) subscription? Who would these people be?

If the answer is "not many," then this is a meaningless thing to freak out about (aside from being mad at Elsevier for extracting rents, fine).

I may not have access to Annals of Endicrinology, but so what? I'm not a doctor, so I couldn't read them and appropriately place them in context anyway.

What remains is the issue of how rents are divided, and if the monopoly power of some publishers blocks the production of knowledge. And price-discrimniation, if Elsevier does that, may in fact help the right quantity get produced anyway...

My guess is that PK is just using this as a canard to discredit refereed journals because he has chosen direct-to-consumer marketing, and now seeks to legitimize it.

"Do we really think there is large proportion of people who (i) can read technical journal articles AND (ii) have no access through their institution to a "free" (in terms of the marginal cost they privately pay to view an article) subscription? Who would these people be?"

If your university library has access to online journals, then they are "free" to you, but of course they are not free to your library, which is paying a very large fee to Elsevier. What if they decide that Elsevier's price is too high, and they stop subscribing? What do you think your university could be doing with all that money they are sending to Elsevier?

I agree--they could use it for lots of things. I just am dubious that the sets of eyes that will pore over a serious article (the ones you--and I--agree) in a serious economics journal will be much expanded relative to now. We might get a new computer, or more summer support, or something. But I doubt that it will expand the "extensive" margins of readers much, and maybe only indirectly if it does.

"They certainly could, but most blogging on economics, including mine, is journalism. "

Journalism in the pejorative sense of the term? Hopefully you're setting your sights higher, since there are some of us who do use this (and other blogs) as learning tools, and have little patience for yellow journalism.

I'm sure stephen will chime in, but my guess is he meant journalism only in the(non-pejorative)sense of describing as clearly as you can an idea in english--something that is clearly valuable. At least that how I view his blog.

But since english is such an imprecise language, and does not allow any quantification of the size of effects, econ blogs are in no way a substitute for what constitutes a really serious analysis of an issue.

Good! Please continue translating your ideas into English for the unwashed and all that. Very helpful. I for one would like it if you referenced your papers more so we could see where and how you connect the theory with the blog posts.

"since english is such an imprecise language"

Yes, case in point the word "journalism", which can have both negative and positive connotations. But as long as terms are properly defined ahead of time and everyone agrees, or as long as they are cleared up along the way, English can be dreadfully accurate.

There is a lot of corruption in the Editorial/Refereeing process. Editors send their friends papers to referees that they know will be sympathetic. Referees are much more likely to accept their friends papers. The American Economic Association needs to audit the acceptance decision by Editors. You could easily see whether an Editor with Minnesota or MIT connections are more likely to use these schools networks for refereeing purposes when the paper comes from someone in the network. There must be statistical tests that can detect bias. Ashenfelter definitely took advantage of the Princeton network for his cronies. Meghuir sent his friends papers to their own coauthors for refereeing.

It's a small world. After you have been in the profession for a while, you know everyone. Everyone is your "friend," more or less. Thus, you're always dealing with friends. If you are an editor, you get papers submitted by friends, and you have them refereed by friends. That's why the refereeing process is anonymous. Plenty of people have had their papers rejected by their friends. Editors get mulitiple reports. When the editor makes a decision, each referee sees the decision letter and the reports of the other referees. If you think your paper has been treated inappropriately, you can write to the journal and complain. At some journals, e.g. the JPE, an editor has to convince one of the other editors before a paper is accepted for publication. There are plenty of checks and balances in the process.

If you're worried about corruption, how can you be guaranteed that the people appointed by the AEA to oversee journals are not going to be corrupt? Maybe they have their own agenda and will use their power to push that agenda with the journals.

The process is not perfect, but it works, for the most part. Try to write better papers. Start your own journal if you're unhappy with what the existing ones publish.

That is not my experience. Knowing editors and referees does matter in a variety of ways. And not everyone knows editors in the same ways. Thus, papers from authors X and Y are judged differently, depending on the connections that X and Y have, where they work, where they went to school, etc.

I wouldn't call it "corruption" but there definitely are networks, clubs, if you will.

If an author is part of the club in a certain subfield, he/she has big advantages in publishing in that subfield.

I agree with Chris, and would like to add that "anonymous" refereeing is not really anonymous. Since most submitted papers have been circulating for a while as working papers it is not that hard for a referee to do a search and find out who the author is.

Any interesting "discussion" going on in the open science blogosphere that might be worth a comment, Dr. Williamson? Say a stimulating dialogue between old adversaries who have been cited by this blog or perhaps contributed to your comments section? Just a passing question...

Because there is no "science," to economics we have all been badly damaged by the shortcomings of the profession.

There is very little integrity and no responsibility.

Take the most simple case: Greenspan. He has testified before Congress that he was totally and completely wrong, to the core, and yet he still can call himself an economist, can collect fees that obviously influence his public statements, etc.

Not only does the "profession" need strong editors, it needs to start defrocking people.

The recent blog war shows that others Lucas, Cochrane, et al, also need to be defrocked, but in this life there are no consequences

No, I did not receive a rejection from a journal this morning. I still want to argue that the peer review system is fundamentally broken in Economics. And Economists should really know better.

Referees write reports that are anonymous to everyone but the editor. They can make claims that are not open to scrutinization. They can abuse the author by asking her to modify paper to suit their agenda (cite them, in particular). They are also not held accountable if they do a lousy job, take forever, or do not care.

Of course, the identity of referees is open to editors. But in many cases, the editors have no way to identify whether referees do a good job, the latter are supposed to be the experts, after all. This is a typical agency problem, where often the principal has little idea of what is going on. The consequence is that journal publications correlate little with quality, and in particular that good works get rejected for flimsy reasons.

The only way out I see is to make referee names public. To some extend this is already done for some institutional working papers, for example at the IMF where the person authorizing the publication is listed on the paper. But this should also apply to rejections. Journals typically acknowledge their referees in one issue a year, but it is difficult to judge them on this.

Rating agencies for financial products are under scrutiny these days. Referees in Economics should as well.

Good luck getting people behind this one. Though you make some VERY fascinating points, youre going to have to do more than bring up a few things that may be different than what weve already heard. What are trying to say here? What do you want us to think? It seems like you cant really get behind a unique thought. Anyway, thats just my opinion.