I'm no big fan of the SCI conference but virtually none of the conferences in computer science fully referee their submissions. A clever student could write a paper with a bogus proof and have a chance of that paper being accepted at a major conference like STOC. I would consider someone who intentionally submits a bogus paper to STOC guilty of academic fraud. Why are these MIT students any different?
Students make mistakes and we should tell them what they did was wrong instead of just glorifying such activities.
The Bogdanoff Affair supports Lance' conjecture that a bogus paper, in principle, might get into STOC.
ReplyDeleteCertainly one might find this behaviour by these students unacceptable, but I don't think one can call it a 'fraud'. There was no element of concealment to the general public; on the contrary, they're probably hoping for as much publicity as possible. A prank - sure, but maybe not fraud (similar thing can be said about the Sokal affair).
ReplyDeleteI think you are being too harsh, Lance. Sending a paper to STOC with a purposely false proof is intently deceitful (the paper is crafted to look real even though the authors know it is bogus), and the motive is purely selfish (an extra paper on the CV). Sending a random paper to the SCI conference is not inherently deceitful (presumably if anyone had looked at the paper they would have realized it was gibberish), and the motive was to expose the low standards of the conference.
ReplyDeleteSo, for example, if someone submitted a paper with a fake proof to STOC, with the intent of admitting it after the fact, and with the motive of showing how shoddy the refereeing process is, I think this would be reasonably legitimate. (Of course, we already know --- and the organizers of STOC would admit --- that the referee process for a conference is not meant to verify all proofs. On the contrary, the organizers of SCI still will not admit that their conference is junk.)
I think I disagree. You say that you do not refree proofs in STOC. However, if the theorem being proved seems counter-intuitive, you *will* read the proof to make sure what's going on. Perhaps you will omit the proofs of mundane theorems, which you intuitively believe to be correct.
ReplyDeleteWhat is different about the SCI affair is that complete and total gibberish was accepted. That is unacceptable by any standards, and the students must be commended on their effort in showing that the conference is a hoax. No self-respecting conference will accept complete gibberish.
Hmmm? Isn't this a bit contradictory? On one side there is CS conferences/journals that claim to be selective, and on the other side there is a refereeeing process that might be flawed (or not existance in the SCI case).
ReplyDeleteIn short, dont blame the mirror for what it shows you. I think the students work in this case should be encouraged.
Lance, I'm glad that for once we totally agree about something. Parody and sarcasm have a place, but that place is not science. Our job as researchers is to do research and write papers, not to expose pretense and idiocy (however much fun that might be). I mean, can you imagine what would have happened if, instead of responding soberly to his critics, Galileo had ridiculed them by putting their arguments into the mouth of a fictional idiot? Or if, instead of trying to replicate Blondlot's N-ray work, Robert Wood had just secretly removed the prism during one of Blondlot's demonstrations? Without a foundation of trust, science would never get anywhere.
ReplyDeleteIsn't the SCI conference a fraud?
ReplyDeleteWhat about RL=L? I take this blog seriously. Over the past couple of decades, I have been moving closer and closer to a proof of the result, and the April 1 announcement very nearly delivered me into a state where I could have consulted the Book proof directly.
ReplyDelete1. Probably, your spam filter works perfectly with SCI's messages (for example, gmail sorts out most of them). SCI and conferences around it continiously reject to remove anyone from their mailing lists (and continue changing their email addresses, evidently, in order to fool spam filters).
ReplyDelete2. SCI is quite different from "normal" CS conference, isn't it? For outsider, it makes bad publicity to CS conferences: I saw several news articles that referred to what's happened as "computer scientists were unable to recognize gibberish". Someone could submit some P=NP paper to SCI, and this would make even worse publicity to our community.
We are not amused. Nobody even read the stupid paper, it got exactly what it deserved! What would be fun is if the paper appears on the authors' resumes and is taken to be representative of their research...
ReplyDelete"Why are these MIT students any different?"
ReplyDeleteIt is so shockingly sad that a well respected computer scientist should (choose to) fail to see the most obvious difference.
It is not about any fine line between fraud and prank, but simply about conducting versus exposing fraud.
I'm not familiar with SCI and would like those who are criticizing it (including those behind this hoax) to elaborate: why is SCI considered a fraud? According to some other comments I've read on this subject, some otherwise well-respected researchers have previously published at SCI.
ReplyDeleteInfact, if you read the hoaxer's website on this (http://pdos.csail.mit.edu/scigen/) you'll see that they never intended fraud at all. Instead, they were hoping to expose (what they felt) were the "low standards" of the WMSCI conference.
However, I believe they failed in their original goal: one paper was rejected outright and the second was only accepted as a "non-reviewed" paper. I'm not 100% sure, but I doubt that a non-peer reviewed paper would have ultimately made it into the published proceedings.
In addition, a follow-up letter (available on their web page) from the conference indicates their reasons for accepting the paper (again, as non-reviewed)--namely, that they hadn't gotten sufficient comments from reviewers to outright reject the paper (though they curiously mention that some reviewers told them they thought the paper was a joke). The status that it was accepted under was such that the authors had complete responsibility for its content, not the conference.
This is not fraud. This is an attempt to deal with spammers by wasting their time. It's no more fraud than the hijinks in The spam letters. Likewise, WMSCI is not a computer science conference, it is a vanity publisher.
ReplyDeleteAnonymouns wrote:the second was only accepted as a "non-reviewed" paper.
ReplyDeleteWell, one could suspect that hardly any paper is reviewed, i.e. the percentage of "non-reviewed" papers is not "very small" but close to hundred.
I'm having some trouble meshing Lance's opinion here with his previous P/NP and the Arts post in which is the quote "I can recognize great music but I can't create great music."
ReplyDeleteSince we can consider the possibility of a complex computation creating art, it is not at all unreasonable to consider such a computation creating scientific papers.
Doesn't this "fraud" simply amount to a demonstration that SCI is (with ~50% accuracy) only capable of recognizing paper validity through a computation equivalent to vocabulary comparison and parsing a CFG? An agrammatical paper without CS buzzwords would probably have been rejected, but that is about where their standards are. To put it another way, the MIT students didn't act as a computationally-powerful "adversary" generating the most misleading paper possible - they actually created a totally random sample of grammatically-connected buzzwords. The probability of this generating a "meaningful" paper (for common sense values of "meaningful") is, intuitively, ZERO. However, for SCI the most-likely probability is around 0.5.
It just so happens that these MIT students (and they're not alone) believe that the standard for publication should be higher - that science is not CFG-complete. Who can blame them? There is no fraud here, simply a demonstration that SCI is an opportunity to present, with 0.5 probability, anything that is at least grammatically correct for a quid pro quo payment of a few hundred dollars.
Since SCI is apparently run independently of academia and as a business, it seems fair that their performance be gauged through harsh real-world antics. As far as I can tell, this is like any "Consumer Reports" review of a service.
I've read on this subject, some otherwise well-respected researchers have previously published at SCI.
ReplyDeleteActually a Google search as suggested returned a while back no well-known names. All the authors were from medium-to-small universities and even in those cases, one wonders if they knew the conference was mickey-mousish. I know of at least one colleague who submitted a paper to a similar conference unknowingly, the paper got accepted and he sent the student to present it. So he never learned how bad the conference was. Several years later some one brought to his attention that the conference was very low quality and this person no longer lists the paper in his resume and he's quite embarrased by the whole thing.
The SCI conference is really just a form of vanity press. If you examine the fees, local arrangements costs, and spam-like form of the e-mail calls for papers it becomes quite apparent that the goal of the conference is not science so much as
ReplyDelete1. money-making and credit for the organizers
2. a publication for the CV of authors who can't get published in other venues
3. an excuse for a paid vacation in a nice location
In that sense the conference itself has aspects of a scam. Some people get duped by scams (or are willing participants for other reasons) so it is certainly possible that legitimate high-quality papers may be sent there. What the MIT students did was expose one aspect of this scam.
The conference got what it deserved.
There are two points i would like to make here. First, SCI is not a scientific endeavor. Publishing there is more akin to buying a doctorate than to doing science. So this maybe is a place for sarcasm and parody. It is also interesting how SCI's review system works. According to their website they have a database of reviewers of which they choose 3 random ones (so that there is no "human intervention"), send the paper to them, and, yes, they do not get reviews usually, i guess. It is obviousy flawed to accept the papers that did not get reviews.
ReplyDeleteThe second point is that the submission and following press response might actualy hurt our field. So the news report Lance links is balanced, but if this gets more publicity people will think that computer scientists write junk papers (funded by government money!) full of gibberish, and cannot even tell their stuff apart from what monkeys type (didn't we suspect that all along? Darn commies!).
Think of the recent "evolution vs. whatever" post. It is a serious point. Everybody in computer science knows stuff like SCI is nonsense, but if the impression is made that accepting pure gibberish at a real conference is easy, too that would be very bad, indeed.
There's an interesting post about this affair in Duh Blog:
ReplyDeletehttp://duhblog.com:8668/space/start/2005-04-15/1#Vanity_Conference,_Fake_Paper,_and_Slanted_Journalism
In my words, while CS scientists think they are making fun of SCI, the general audience is making fun of CS scientists.
Although the SCI conference may be have low standards, this not really related to the bogus paper. The organizers of the conference claim that they received no review comments (presumably not even from PC members) about the bogus paper and then accepted it as a "non-reviewed paper". This seems to me like getting a bogus paper on the arXiv.org e-Print archive. It may be possible to fool an arXiv moderator, but so what.
ReplyDeleteThe criticism of SCI that I think is implied by the bogus paper is mainly procedural: the PC chair could not ensure that every submission was looked at by at least one person.
Yes, it's the same as arXiv except for 1) lack of peer-moderation, 2) the $300 registration fee.
ReplyDeleteIt is reasonable to think that if SCI were at all a legitimate effort, the $300 would go to moderation. It clearly does not.
I very strongly opppose the previous post concerning the archive compared to SCI. Putting a paper on the arxiv is not a publication. It is more like what the old technical report was. While there is moderation now, no one checks validity. Still I think that the MIT-prank paper would not have gotten into the arXiv. This is because a new author has to be "endorsed" now, which would not have happened with this paper. So, while there is actually no quality standard on the ArcIve, it is probably more restricted than SCI.
ReplyDeleteAfter reading all the comments on this post, two things are clear to me:
ReplyDelete1. The SCI conference is a scam.
2. The MIT students did not commit fraud because their deception was intended to be caught immediately by the refereeing process purportedly used at SCI.
To elaborate on 2: It was SCI who defined the terms of submission. In the context of those terms, the students' means of exposing SCI involved zero deception. One can think of the students as verifiers who use coin tosses (literally) to catch cheating provers (SCI scammers). The protocol of interaction was established ahead of time by SCI and then SCI was caught cheating using only that protocol.
If anything, the actions of SCI are *MUCH* closer to what I would call "fraud" than the actions of these students.
In this age of spam and deceit in the name of profit, an action that exposes shameless scammers without resorting to deceit is to be commended.
It is unfortunate that the public misinterprets this exposition as bungling, but the error of the public does not decrease the merits of the action. If we scientists pandered to the public's misconceptions in order to convince them that our work is worth their tax money then we would be little more than scammers ourselves.
Scott Aaronson spoke of a foundation of trust, but he only applied his argument towards the submitters. What about SCI? Many scientists trusted SCI to be a legitimate conference only to discover years later that publishing in that conference is an embarassment. SCI's violation of trust was far greater than that of the students (if their actions can even be called a violation of trust, which I claim they cannot).
Gus
To elaborate on 2: It was SCI who defined the terms of submission. In the context of those terms, the students' means of exposing SCI involved zero deception. One can think of the students as verifiers who use coin tosses (literally) to catch cheating provers (SCI scammers). The protocol of interaction was established ahead of time by SCI and then SCI was caught cheating using only that protocol.
ReplyDeleteThis would be more compelling if the conference had claimed their decision was based on a review of the paper. The conference openly admits that they have a "non-reviewed" category, where anything goes. A stupid category, but technically not fraud. The prover isn't asserting x is in L (or
not in L), so they were not "caught cheating".
I am not claiming that the conference is good, just disagreeing with the previously posted analysis.
Quoting Lance
ReplyDeleteWithout a foundation of trust, science would never get anywhere.
Sounds quite religious, doesn't it?
Gosh ... does the fact that people apparently didn't get my joke argue for or against the point I was trying to make?
ReplyDeleteScott: maybe you should've added to your post a Homer-like disclaimer as in "Oh, by the way, I was being sarcastic."
ReplyDeleteAnyway, I really enjoyed your post, and the fact that several people misunderstood it makes it even more epigrammatic.
Oh man, and I thought for a moment that I'd actually written a decent comment (despite the lame comparison to interactive proofs). That's the last time I sign my name to anything in this blog... :(
ReplyDeleteGus
I think what the students did is not necessary to show the lack of standards of the conference. There is already an unofficial evaluation of conferences, and is well know that the SCI conferences rank pretty low. Mockery, on the other hand, brings only disadvantages for the field -- with the additional side effect that now the chance of having good things coming from these conferences will become close to zero.
ReplyDeleteQuery to Lance Fortnow: Hope to see reply to this:
ReplyDeleteIsn't it a good idea to submit the papers on archives and shouldn't the leading computer scientists like yourself promote it?
Aaronson said
ReplyDeleteGosh ... does the fact that people apparently didn't get my joke argue for or against the point I was trying to make?
Dude, that was -way- too subtle. The only possible giveaway was the Galileo example, which only sparked a tenuous memory ("didn't Galilleo actually do that? or maybe Simplicio (or was it Stultio? or shoot was that by Hume?) To expose my own ignorance, who the heck is Blondlot or Wood?). The rest was reasonable enough to be taken seriously. As to the "foundation of trust" thingy, OK, that's off, but it just doesn't sound like you're kidding.
Oh... was that your point? Sorta like meta-joking? ;)
Fraud implies some serious intent to "get away with something", to pass one thing off as another. Anyone who spent 2 minutes reading their paper would immediately realize that this isn�t a real paper. They might be confused regarding the intent, but they would know instantly and absolutely that the authors weren�t even trying to fool anyone into thinking that it was a real piece of science. The papers that SCIgen churns out are about as subtle as a Three Stooges film, and that�s not fraud, that�s parody. Parody is a crucial tool in the toolkit of those looking to expose shams and shallow thinking, and as such I think the students� actions were completely reasonable.
ReplyDelete