tag:blogger.com,1999:blog-3722233.post1136782380308825005..comments2024-03-27T19:58:17.387-05:00Comments on Computational Complexity: You Can't Separate the Art from the ArtistLance Fortnowhttp://www.blogger.com/profile/06752030912874378610noreply@blogger.comBlogger73125tag:blogger.com,1999:blog-3722233.post-20837497114677926512009-03-06T12:48:00.000-06:002009-03-06T12:48:00.000-06:00Also, it may not be necessary to note papers with ...<I>Also, it may not be necessary to note papers with errors *if* all the authors have corrected versions (noting the error/difference with originally published theorems so that people don't use the wrong theorems) available on their webpages.<BR/></I><BR/><BR/>My inclination would be to note it anyway (with a link to the correction), for two reasons. One, I'd like to know statistics on how often serious errors occur, and that requires a reasonably comprehensive list. Two, I think public shaming plays a role here. Anybody can make rare mistakes, but some people would show up on this list much more often than others and we ought to see the full list (rather than letting people cover up their mistakes if they manage to correct them later). We should certainly distinguish between fixable and unfixable errors, but even fixable errors should still embarrass the authors at least a little.<BR/><BR/><I>That would be one of the benefits of putting all papers on arxiv and having blog-like discussions (similar to public reviews) about them. </I><BR/><BR/>That could help, but regardless I would still love to have an overall list in addition (for the reasons above). Plus, I bet most of the blog-like discussions would either have no posts or rapidly degenerate into chaos. A first step would be to require the use of real names and filter out all cranks and amateurs, but even that might not be enough to ensure a good discussion, while it might already be a high enough barrier that few people would want to participate.<BR/><BR/><I>It could also just be a website where people can post anonymously.</I><BR/><BR/>Yes, but I'd like some reputable person to moderate everything. (For example, to filter out stupid assertions, to contact authors so they are aware their work is being discussed, etc.) Several times, cranks have written to me to tell me my work has fundamental errors in it, with no basis in fact, and I bet that unmoderated anonymous submissions would include a lot of similar nonsense.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-48015493696707684372009-03-06T11:33:00.000-06:002009-03-06T11:33:00.000-06:00"Could some brave person with tenure (i.e., not me..."Could some brave person with tenure (i.e., not me) please start maintaining a public list of bogus FOCS/STOC/SODA papers?"<BR/><BR/>It could also just be a website where people can post anonymously. Also, it may not be necessary to note papers with errors *if* all the authors have corrected versions (noting the error/difference with originally published theorems so that people don't use the wrong theorems) available on their webpages.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-79339623526554629512009-03-06T11:29:00.000-06:002009-03-06T11:29:00.000-06:00That would be one of the benefits of putting all p...That would be one of the benefits of putting all papers on arxiv and having blog-like discussions (similar to public reviews) about them. Right now, on ECCC, you can submit a comment, but usually it is a simplification or correction and it is very "formal". It would be better to have an arxiving website in which people can informally discuss a paper and note/ask about errors.<BR/><BR/>Then retractions could also be posted with the paper and a person wanting to reference a paper could look at the discussion and all follow up corrections, etc., which would be in one place.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-51984044460846259122009-03-06T10:46:00.000-06:002009-03-06T10:46:00.000-06:00These are all cases in which the authors publicly ...<I>These are all cases in which the authors publicly stated retractions of their main theorems, but for each paper, at least some subset of the authors has the original (unrevised) paper listed or available on the webpage without noting any error.</I><BR/><BR/>Could some brave person with tenure (i.e., not me) please start maintaining a public list of bogus FOCS/STOC/SODA papers? It would be great to have a bunch of lists: retractions by the authors due to fatal errors, nontrivial but correctable errors with information on the corrections needed, controversial or debated proofs (including criticisms of the paper and a response from the authors if they deny the error), plagiarized papers, inadvertent rediscovery of previously known results, etc.<BR/><BR/>I don't know of any plagiarized papers in major conferences but I'd be very interested to find out if somebody else knows about some. All the other categories are real and it would be a great service to the community to make this information available.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-24549544656837566952009-03-06T09:03:00.000-06:002009-03-06T09:03:00.000-06:00Anon 67: we don't have to write software from scra...Anon 67: we don't have to write software from scratch, we just need to modify an existing conference management package. I'd imagine that the needed modifications would take someone who already knows the relevant code only a day or two. The code modification would be a minor part of the cost of running an experiment.Unknownhttps://www.blogger.com/profile/01106301822827737278noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-19665758721920548552009-03-05T21:28:00.000-06:002009-03-05T21:28:00.000-06:00"To Anon 63:Do you have some specific examples of ..."To Anon 63:<BR/>Do you have some specific examples of recent papers that contain an error? Or pointers to author's websites listing the errata?"<BR/><BR/>A randomness-efficient sampler for matrix-valued functions and applications. FOCS 2005<BR/><BR/>Improved Embedding of Graph Metrics into Random Trees. SODA 2006<BR/><BR/>Vickrey Prices and Shortest Paths: What is an edge worth?. FOCS 2001<BR/><BR/>These are all cases in which the authors publicly stated retractions of their main theorems, but for each paper, at least some subset of the authors has the original (unrevised) paper listed or available on the webpage without noting any error.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-50080338160483702852009-03-05T18:26:00.000-06:002009-03-05T18:26:00.000-06:00Warren:Are you willing to write such software?Warren:<BR/><BR/>Are you willing to write such software?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-35546125150257564392009-03-05T01:37:00.000-06:002009-03-05T01:37:00.000-06:00To Anon 63:Do you have some specific examples of r...To Anon 63:<BR/>Do you have some specific examples of recent papers that contain an error? Or pointers to author's websites listing the errata?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-44914086806188413362009-03-04T20:26:00.000-06:002009-03-04T20:26:00.000-06:00Anon 11:51 AM, March 04, 2009 wrote:The first comm...Anon 11:51 AM, March 04, 2009 wrote:<BR/><I>The first commenter has pointed out that Kathryn's experience is that DBR is essentially incompatible with having individual PC members send out papers for expert subreviews.</I><BR/><BR/>Surely we're clever enough to design procedures enabling DBR and subreviews to coexist.<BR/><BR/>One approach is to simply ignore conflicts of interest when sending papers to subreviewers. If the subreviewer recognizes that they have a conflict of interest they are required to return the paper, otherwise does it matter if the reviewer has a conflict if they don't know it? I know someone who was asked to subreview his own paper for AAAI once, so perhaps AAAI takes this approach.<BR/><BR/>The specialization required to evaluate theory papers would probably lead to an unacceptable rate of accidental conflicts if the above approach were used, so we'd probably need to use a more pro-active approach. For example:<BR/><BR/>1) Ask submitting authors to submit a list of people who should not review their paper. An approximate list could be populated with names scraped from the author websites, CVs, DBLP or similar.<BR/><BR/>2) PC members indicate at least two possible subreviewers for each paper.<BR/><BR/>3) The conference management software selects a subreviewer for each PC member taking into account:<BR/>* Several PC members cannot be assigned the same subreviewer for the same paper<BR/>* Conflicts of interest<BR/>* Whether the subreviewer agrees to do a subreview<BR/>* A little randomization to make it harder for the PC to deduce the author list from which subreviewers get selected<BR/><BR/>The above proposal may sound complicated, but with automation (we're computer scientists after all!) it shouldn't add much overhead.Unknownhttps://www.blogger.com/profile/01106301822827737278noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-43792395298336450802009-03-04T19:24:00.000-06:002009-03-04T19:24:00.000-06:00Can you point to a single link given here to a stu...<I>Can you point to a single link given here to a study suggesting DBR is a bad idea?</I><BR/><BR/>Here's a link that (I believe) hasn't been mentioned here yet:<BR/><BR/>http://cmsprod.bgu.ac.il/NR/rdonlyres/D5BB74A3-0AB5-4237-AA1D-E0039B0547EF/66706/Buddenreplyonwhittakerfemaleauthorship3.pdf<BR/><BR/>It argues that DBR does not in fact increase the representation of female authors (contrary to the results of a previous study using a more limited data set) and suggests not using DBR.<BR/><BR/><I>I didn't think so.</I><BR/><BR/>I don't go around citing studies since I don't think any of them are really compelling. Nobody has clean, definitive data that suffice to resolve the issue, so the conclusions will always be debatable. Considering that the authors always have an agenda and never have great data, I think it's silly to put your faith in any of them.<BR/><BR/>But nobody should think there aren't anti-DBR studies. If the studies all agreed, we wouldn't be having this discussion. The fact that they don't agree is actually evidence of the weakness of the data. I don't think people are so intellectually dishonest that they wouldn't come to agreement given sufficiently compelling data. On the other hand, I see no feasible way to get such data, so I suspect the issue will remain contentious for a long time.<BR/><BR/><I>The idea that name matters because famous people don't make errors is pretty annoying.</I><BR/><BR/>Agreed. That's why nobody is making such a claim.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-32060002252821585012009-03-04T17:07:00.000-06:002009-03-04T17:07:00.000-06:00The idea that name matters because famous people d...The idea that name matters because famous people don't make errors is pretty annoying. There have been some prominent examples of errors in major conferences in the past years. Some people note it on their webpage and other people just list the paper as a publication anyway without acknowledgement.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-77737686861891319762009-03-04T16:13:00.000-06:002009-03-04T16:13:00.000-06:00You write as if there's a burden on opponents of D...<I>You write as if there's a burden on opponents of DBR to refute it, but the situation is quite symmetrical: to refute SBR what needs to be shown is that it in fact makes the system as a whole worse off.</I><BR/><BR/>True enough, and quite a few studies have already been linked to by the proponents of SBR in these threads (see Sorelle's or Suresh's blog). Can you point to a single link given here to a <I>study</I> suggesting DBR is a bad idea? I didn't think so.<BR/><BR/>So you see the situation is not symmetric at all. The ball is now on the SBR proponents court, and made-up extreme examples such as proofs by Lance of P=NP don't illuminate things, I'm sorry to say.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-57271427070494115032009-03-04T12:04:00.000-06:002009-03-04T12:04:00.000-06:00Isn't it funny how so many opponents of double-bli...<I>Isn't it funny how so many opponents of double-blind reviewing are posting their comments anonymously? (A discussion on this thread is far less consequential than the submission of a paper to FOCS/STOC, yet they don't want to be held to account.)</I><BR/><BR/>What exactly would being "held to account" entail, and why should anyone want it?<BR/><BR/>If I claimed to be a real expert in how conference reviewing should be done, and wanted everyone to accept my authority, it would be reasonable to ask who I was and how I came to develop this supposed expertise. Clearly I'm not a real expert, but then again neither is anybody else involved in this discussion, so I guess we're all on an equal footing.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-14146100489336705702009-03-04T11:51:00.000-06:002009-03-04T11:51:00.000-06:00Suresh on his blog points to an essay by Kathryn M...Suresh on his <A HREF="http://geomblog.blogspot.com/2009/03/dbr-moving-forward.html" REL="nofollow">blog</A> points to an <A HREF="http://www.cs.utexas.edu/users/mckinley/notes/blind.html" REL="nofollow">essay by Kathryn McKinley</A> advocating the benefits of DBR and explaining the best way to use DBR based on experience. The first commenter has pointed out that Kathryn's experience is that DBR is essentially incompatible with having individual PC members send out papers for expert subreviews. Expert subreviews are the lifeblood of all the major theory conference PCs. I can't imagine doing away with such sub-reviews - a significant portion of the value of the expertise of PC members is in finding the right experts for subreviews. Should we really do away with this and replace it with DBR?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-78935924624996051432009-03-04T11:43:00.000-06:002009-03-04T11:43:00.000-06:00As a member of the networking community I thought ...<I>As a member of the networking community I thought I'd share a bit of an outsider's perspective.<BR/></I><BR/>...<BR/><I><BR/>When I'm reviewing a paper I'd say at least 75% of the time I have zero idea who wrote the paper</I><BR/><BR/>I suspect this has a lot to do with the size and breadth of the networking community. When I review Crypto papers, I often can't specify the exact list of authors (unless I happen to have read a preprint or seen a talk, which is not uncommon for the better papers). However, for a majority of papers I have a pretty good idea of who wrote them. In other words, I can identify at least one of the authors, or I can tell that X and/or X's students are involved, or I can narrow it down to one of a couple of people who sometimes collaborate. It's pretty rare for me to have absolutely no idea who wrote a paper, and I don't think it has ever happened (in 5+ years of reviewing) for a paper involving prestigious researchers or institutions.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-87386650344643239032009-03-04T11:41:00.000-06:002009-03-04T11:41:00.000-06:00Isn't it funny how so many opponents of double-bli...Isn't it funny how so many opponents of double-blind reviewing are posting their comments anonymously? (A discussion on this thread is far less consequential than the submission of a paper to FOCS/STOC, yet they don't want to be held to account.)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-74324899396864809602009-03-04T11:29:00.001-06:002009-03-04T11:29:00.001-06:00People who are against DBR keep on giving isolated...<I>People who are against DBR keep on giving isolated examples where DBR would be bad. A single example in no way refutes DBR. </I><BR/><BR/>The "isolated" examples are actually patterns that occur some fraction of the time. For most papers (i.e., more than 50%), SBR vs. DBR should have no effect - it would be amazing and unsettling if a majority of decisions were changed. So we are automatically talking about a minority of all papers, and probably a fairly small minority. Dismissing these cases as "isolated" is tantament to saying the whole issue doesn't matter, in which case we might as well stick with the system we've got.<BR/><BR/><I>I can come up with a similar example where DBR would be great: a well known crank who has submitted dozens of incorrect P=NP proofs finally gets it right. Under SBR paper doesn't even get read, under DBR proof is accepted and the world becomes a better place.</I><BR/><BR/>I assume this is a joke. If someone discovered a credible proof of P=NP, do you really think they would keep it secret throughout the anonymous reviewing process? Even if they tried, I bet word would leak out.<BR/><BR/>In any case, DBR won't help cranks. It's not so easy to write a paper that can blend in with professional work (it takes a lot of knowledge in the field as well as understanding of professional norms and customs), and I've never seen a crank do it.<BR/><BR/><I>AS you can see, to refute DBR what needs to be shown is that the system as a whole would be worse off.</I><BR/><BR/>You write as if there's a burden on opponents of DBR to refute it, but the situation is quite symmetrical: to refute SBR what needs to be shown is that it in fact makes the system as a whole worse off.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-75928493006058238372009-03-04T11:29:00.000-06:002009-03-04T11:29:00.000-06:00As a member of the networking community I thought ...As a member of the networking community I thought I'd share a bit of an outsider's perspective.<BR/><BR/>No the system we have for conferences like Sigcomm isn't perfect. But double-blind reviewing has two effects. 1. When I'm reviewing a paper I'd say at least 75% of the time I have zero idea who wrote the paper (another 20% of the time I have only a vague guess) - so the paper's authorship can't affect the outcome. 2. The blind nature of the review serves as a subtle reminder to me as the reviewer that the task at hand is a sacred oath and that it should be performed with the utmost care and impartiality.<BR/><BR/>Yes, we do get complaints of <A HREF="http://en.wikipedia.org/wiki/Ingroup_bias" REL="nofollow">ingroup bias</A>, but I'd argue that the bias we show is towards particular types of work rather than particular authors. Everyone can submit papers on topics similar to those of, say, Ion Stoica. Not everyone can change their name to Ion Stoica. So bias towards certain types of work is far more egalitarian.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-22302553394845060132009-03-04T11:03:00.000-06:002009-03-04T11:03:00.000-06:00For those who claim to disagree completely with La...<I>For those who claim to disagree completely with Lance, answer this...</I><BR/><BR/>People who are against DBR keep on giving isolated examples where DBR would be bad. A single example in no way refutes DBR. <BR/><BR/>I can come up with a similar example where DBR would be great: a well known crank who has submitted dozens of incorrect P=NP proofs finally gets it right. Under SBR paper doesn't even get read, under DBR proof is accepted and the world becomes a better place.<BR/><BR/>AS you can see, to refute DBR what needs to be shown is that the system as a whole would be worse off.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-83870512610661371862009-03-04T09:39:00.000-06:002009-03-04T09:39:00.000-06:00For those who claim to disagree completely with La...For those who claim to disagree completely with Lance, answer this: would you be any more likely to read a paper on the arXiv claiming P \neq NP if the author were Fortnow than you would if the author were someone you never heard of? Do you think a reviewer should not spend more time reading the former than the latter?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-79569288196658358122009-03-04T06:55:00.000-06:002009-03-04T06:55:00.000-06:00I wonder if what Aaron said is interesting. I'll n...I wonder if what Aaron said is interesting. I'll never know since I only read opinions of people who prove their mettle by having an NSF grant larger than mine.<BR/><BR/>What could be wrong with that?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-46279688368216566582009-03-04T01:34:00.000-06:002009-03-04T01:34:00.000-06:00A lot of the comments have a different underlying ...<I>A lot of the comments have a different underlying flavor to me: anger about being shut out of having a certain reputation, of not being able to stand on the same podium as a Goedel Prize winner, or a Nevanlinna Prize winner.</I><BR/><BR/>Another one of the canards usually brought up when discussing improvements to the current system: it's all sour grapes. <BR/><BR/><I>Are the "bloggers" who deride any other conference likely to be on the hiring committee that is considering you?</I><BR/><BR/>Actually yes, and in your grant awarding committees as well.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-5907529770537763032009-03-03T22:38:00.000-06:002009-03-03T22:38:00.000-06:00STOC/FOCS/SODA are broken on so many different cou...STOC/FOCS/SODA are broken on so many different counts, let's not even get started. I have had two different reviews to the same submission, 1st reviewer - "The nice aspect of this paper is that the proofs are simple. However, the presentation of the paper can be improved significantly". 2nd reviewer - "The paper is very well written. The only disadvantage is that the proofs are very simple."Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-87966741223727829822009-03-03T22:32:00.000-06:002009-03-03T22:32:00.000-06:00Agreed. Definitely SODA/STOC/FOCS and other confer...Agreed. Definitely SODA/STOC/FOCS and other conferences will become more meaningful with DBR. I see several people arguing as to whether DBR will truly be DBR or not. That is a totally different point. Even if DBR is not truly "double-blind", it is still better than not having DBR at all.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-91281312038765539362009-03-03T22:04:00.000-06:002009-03-03T22:04:00.000-06:00One thing is sure: the review process in FOCS/STOC...One thing is sure: the review process in FOCS/STOC/SODA is broken.<BR/>It encourages anything else but advancing the field!!<BR/>The reputation is much more important than the quality of the papers. That's really unfortunate!!Anonymousnoreply@blogger.com