Computational Complexity and other fun stuff in math and computer science from Lance Fortnow and Bill Gasarch

Thursday, April 21, 2005

A Fine Line Between Prank and Fraud

You have probably heard this story by now. Some MIT students created a
computer-generated paper accepted to a non-reviewed session of the
Systemics, Cybernetics and Informatics conference. I haven't mentioned
the story earlier because I didn't want to give the students extra
publicity but now that the story has hit
the AP wire something needs to be said: What these students did
was just plain wrong.

I'm no big fan of the SCI conference but virtually none of the
conferences in computer science fully referee their submissions. A clever
student could write a paper with a bogus proof and have a chance of
that paper being accepted at a major conference like STOC. I would
consider someone who intentionally submits a bogus paper to STOC
guilty of academic fraud. Why are these MIT students any different?

Students make mistakes and we should tell them what they did was wrong
instead of just glorifying such activities.

33 comments:

Certainly one might find this behaviour by these students unacceptable, but I don't think one can call it a 'fraud'. There was no element of concealment to the general public; on the contrary, they're probably hoping for as much publicity as possible. A prank - sure, but maybe not fraud (similar thing can be said about the Sokal affair).

I think you are being too harsh, Lance. Sending a paper to STOC with a purposely false proof is intently deceitful (the paper is crafted to look real even though the authors know it is bogus), and the motive is purely selfish (an extra paper on the CV). Sending a random paper to the SCI conference is not inherently deceitful (presumably if anyone had looked at the paper they would have realized it was gibberish), and the motive was to expose the low standards of the conference.

So, for example, if someone submitted a paper with a fake proof to STOC, with the intent of admitting it after the fact, and with the motive of showing how shoddy the refereeing process is, I think this would be reasonably legitimate. (Of course, we already know --- and the organizers of STOC would admit --- that the referee process for a conference is not meant to verify all proofs. On the contrary, the organizers of SCI still will not admit that their conference is junk.)

I think I disagree. You say that you do not refree proofs in STOC. However, if the theorem being proved seems counter-intuitive, you *will* read the proof to make sure what's going on. Perhaps you will omit the proofs of mundane theorems, which you intuitively believe to be correct.

What is different about the SCI affair is that complete and total gibberish was accepted. That is unacceptable by any standards, and the students must be commended on their effort in showing that the conference is a hoax. No self-respecting conference will accept complete gibberish.

Hmmm? Isn't this a bit contradictory? On one side there is CS conferences/journals that claim to be selective, and on the other side there is a refereeeing process that might be flawed (or not existance in the SCI case).

In short, dont blame the mirror for what it shows you. I think the students work in this case should be encouraged.

Lance, I'm glad that for once we totally agree about something. Parody and sarcasm have a place, but that place is not science. Our job as researchers is to do research and write papers, not to expose pretense and idiocy (however much fun that might be). I mean, can you imagine what would have happened if, instead of responding soberly to his critics, Galileo had ridiculed them by putting their arguments into the mouth of a fictional idiot? Or if, instead of trying to replicate Blondlot's N-ray work, Robert Wood had just secretly removed the prism during one of Blondlot's demonstrations? Without a foundation of trust, science would never get anywhere.

What about RL=L? I take this blog seriously. Over the past couple of decades, I have been moving closer and closer to a proof of the result, and the April 1 announcement very nearly delivered me into a state where I could have consulted the Book proof directly.

1. Probably, your spam filter works perfectly with SCI's messages (for example, gmail sorts out most of them). SCI and conferences around it continiously reject to remove anyone from their mailing lists (and continue changing their email addresses, evidently, in order to fool spam filters).

2. SCI is quite different from "normal" CS conference, isn't it? For outsider, it makes bad publicity to CS conferences: I saw several news articles that referred to what's happened as "computer scientists were unable to recognize gibberish". Someone could submit some P=NP paper to SCI, and this would make even worse publicity to our community.

We are not amused. Nobody even read the stupid paper, it got exactly what it deserved! What would be fun is if the paper appears on the authors' resumes and is taken to be representative of their research...

I'm not familiar with SCI and would like those who are criticizing it (including those behind this hoax) to elaborate: why is SCI considered a fraud? According to some other comments I've read on this subject, some otherwise well-respected researchers have previously published at SCI.

Infact, if you read the hoaxer's website on this (http://pdos.csail.mit.edu/scigen/) you'll see that they never intended fraud at all. Instead, they were hoping to expose (what they felt) were the "low standards" of the WMSCI conference.

However, I believe they failed in their original goal: one paper was rejected outright and the second was only accepted as a "non-reviewed" paper. I'm not 100% sure, but I doubt that a non-peer reviewed paper would have ultimately made it into the published proceedings.

In addition, a follow-up letter (available on their web page) from the conference indicates their reasons for accepting the paper (again, as non-reviewed)--namely, that they hadn't gotten sufficient comments from reviewers to outright reject the paper (though they curiously mention that some reviewers told them they thought the paper was a joke). The status that it was accepted under was such that the authors had complete responsibility for its content, not the conference.

This is not fraud. This is an attempt to deal with spammers by wasting their time. It's no more fraud than the hijinks in The spam letters. Likewise, WMSCI is not a computer science conference, it is a vanity publisher.

I disagree, Lance. Calling the MIT students' prank �academic fraud� presumes that (1) the conference has some academic value and (2) that the prank was designed to mislead. I don't think either of those conditions is valid here. Their paper was *transparently* bogus.

If someone submitted such an obviously fake paper to STOC, the program committee would have a good laugh, someone on the PC with a sense of humor and some free time might write some obviously fake reviews, the paper would be rejected, and that would be the end of it.

I'm having some trouble meshing Lance's opinion here with his previous P/NP and the Arts post in which is the quote "I can recognize great music but I can't create great music."

Since we can consider the possibility of a complex computation creating art, it is not at all unreasonable to consider such a computation creating scientific papers.

Doesn't this "fraud" simply amount to a demonstration that SCI is (with ~50% accuracy) only capable of recognizing paper validity through a computation equivalent to vocabulary comparison and parsing a CFG? An agrammatical paper without CS buzzwords would probably have been rejected, but that is about where their standards are. To put it another way, the MIT students didn't act as a computationally-powerful "adversary" generating the most misleading paper possible - they actually created a totally random sample of grammatically-connected buzzwords. The probability of this generating a "meaningful" paper (for common sense values of "meaningful") is, intuitively, ZERO. However, for SCI the most-likely probability is around 0.5.

It just so happens that these MIT students (and they're not alone) believe that the standard for publication should be higher - that science is not CFG-complete. Who can blame them? There is no fraud here, simply a demonstration that SCI is an opportunity to present, with 0.5 probability, anything that is at least grammatically correct for a quid pro quo payment of a few hundred dollars.

Since SCI is apparently run independently of academia and as a business, it seems fair that their performance be gauged through harsh real-world antics. As far as I can tell, this is like any "Consumer Reports" review of a service.

I've read on this subject, some otherwise well-respected researchers have previously published at SCI.

Actually a Google search as suggested returned a while back no well-known names. All the authors were from medium-to-small universities and even in those cases, one wonders if they knew the conference was mickey-mousish. I know of at least one colleague who submitted a paper to a similar conference unknowingly, the paper got accepted and he sent the student to present it. So he never learned how bad the conference was. Several years later some one brought to his attention that the conference was very low quality and this person no longer lists the paper in his resume and he's quite embarrased by the whole thing.

The SCI conference is really just a form of vanity press. If you examine the fees, local arrangements costs, and spam-like form of the e-mail calls for papers it becomes quite apparent that the goal of the conference is not science so much as

1. money-making and credit for the organizers 2. a publication for the CV of authors who can't get published in other venues 3. an excuse for a paid vacation in a nice location

In that sense the conference itself has aspects of a scam. Some people get duped by scams (or are willing participants for other reasons) so it is certainly possible that legitimate high-quality papers may be sent there. What the MIT students did was expose one aspect of this scam.

There are two points i would like to make here. First, SCI is not a scientific endeavor. Publishing there is more akin to buying a doctorate than to doing science. So this maybe is a place for sarcasm and parody. It is also interesting how SCI's review system works. According to their website they have a database of reviewers of which they choose 3 random ones (so that there is no "human intervention"), send the paper to them, and, yes, they do not get reviews usually, i guess. It is obviousy flawed to accept the papers that did not get reviews.

The second point is that the submission and following press response might actualy hurt our field. So the news report Lance links is balanced, but if this gets more publicity people will think that computer scientists write junk papers (funded by government money!) full of gibberish, and cannot even tell their stuff apart from what monkeys type (didn't we suspect that all along? Darn commies!).

Think of the recent "evolution vs. whatever" post. It is a serious point. Everybody in computer science knows stuff like SCI is nonsense, but if the impression is made that accepting pure gibberish at a real conference is easy, too that would be very bad, indeed.

Although the SCI conference may be have low standards, this not really related to the bogus paper. The organizers of the conference claim that they received no review comments (presumably not even from PC members) about the bogus paper and then accepted it as a "non-reviewed paper". This seems to me like getting a bogus paper on the arXiv.org e-Print archive. It may be possible to fool an arXiv moderator, but so what.

The criticism of SCI that I think is implied by the bogus paper is mainly procedural: the PC chair could not ensure that every submission was looked at by at least one person.

I very strongly opppose the previous post concerning the archive compared to SCI. Putting a paper on the arxiv is not a publication. It is more like what the old technical report was. While there is moderation now, no one checks validity. Still I think that the MIT-prank paper would not have gotten into the arXiv. This is because a new author has to be "endorsed" now, which would not have happened with this paper. So, while there is actually no quality standard on the ArcIve, it is probably more restricted than SCI.

After reading all the comments on this post, two things are clear to me:

1. The SCI conference is a scam.

2. The MIT students did not commit fraud because their deception was intended to be caught immediately by the refereeing process purportedly used at SCI.

To elaborate on 2: It was SCI who defined the terms of submission. In the context of those terms, the students' means of exposing SCI involved zero deception. One can think of the students as verifiers who use coin tosses (literally) to catch cheating provers (SCI scammers). The protocol of interaction was established ahead of time by SCI and then SCI was caught cheating using only that protocol.

If anything, the actions of SCI are *MUCH* closer to what I would call "fraud" than the actions of these students.

In this age of spam and deceit in the name of profit, an action that exposes shameless scammers without resorting to deceit is to be commended.

It is unfortunate that the public misinterprets this exposition as bungling, but the error of the public does not decrease the merits of the action. If we scientists pandered to the public's misconceptions in order to convince them that our work is worth their tax money then we would be little more than scammers ourselves.

Scott Aaronson spoke of a foundation of trust, but he only applied his argument towards the submitters. What about SCI? Many scientists trusted SCI to be a legitimate conference only to discover years later that publishing in that conference is an embarassment. SCI's violation of trust was far greater than that of the students (if their actions can even be called a violation of trust, which I claim they cannot).

To elaborate on 2: It was SCI who defined the terms of submission. In the context of those terms, the students' means of exposing SCI involved zero deception. One can think of the students as verifiers who use coin tosses (literally) to catch cheating provers (SCI scammers). The protocol of interaction was established ahead of time by SCI and then SCI was caught cheating using only that protocol.

This would be more compelling if the conference had claimed their decision was based on a review of the paper. The conference openly admits that they have a "non-reviewed" category, where anything goes. A stupid category, but technically not fraud. The prover isn't asserting x is in L (or not in L), so they were not "caught cheating".

I am not claiming that the conference is good, just disagreeing with the previously posted analysis.

I think what the students did is not necessary to show the lack of standards of the conference. There is already an unofficial evaluation of conferences, and is well know that the SCI conferences rank pretty low. Mockery, on the other hand, brings only disadvantages for the field -- with the additional side effect that now the chance of having good things coming from these conferences will become close to zero.

Gosh ... does the fact that people apparently didn't get my joke argue for or against the point I was trying to make?

Dude, that was -way- too subtle. The only possible giveaway was the Galileo example, which only sparked a tenuous memory ("didn't Galilleo actually do that? or maybe Simplicio (or was it Stultio? or shoot was that by Hume?) To expose my own ignorance, who the heck is Blondlot or Wood?). The rest was reasonable enough to be taken seriously. As to the "foundation of trust" thingy, OK, that's off, but it just doesn't sound like you're kidding.

Fraud implies some serious intent to "get away with something", to pass one thing off as another. Anyone who spent 2 minutes reading their paper would immediately realize that this isn�t a real paper. They might be confused regarding the intent, but they would know instantly and absolutely that the authors weren�t even trying to fool anyone into thinking that it was a real piece of science. The papers that SCIgen churns out are about as subtle as a Three Stooges film, and that�s not fraud, that�s parody. Parody is a crucial tool in the toolkit of those looking to expose shams and shallow thinking, and as such I think the students� actions were completely reasonable.