Wednesday, November 07, 2007

Resubmitting Rejected FOCS paper to STOC

(Guest post by Kamal Jain Resubmitting the rejected papers from FOCS to STOC?

If your paper was rejected by FOCS and you're submitting it to STOC, here are my thoughts on how you can increase your chances of acceptance. Given the low acceptance rate for FOCS, I am sure many of us will be resubmitting our rejected papers to STOC. Many of us will be incorporating the FOCS PC comments. And there's also a realistic chance that FOCS PC misunderstood our papers. So what should we do so that STOC committee does not repeat the misunderstanding?

Let me first describe the general methods to contain the chances of misunderstanding. I will then describe why the chance of misunderstanding has increased for STOC PC on resubmitted papers by giving you an insider view of FOCS 2007 PC. We can then discuss what we could do to minimize that.

Contain the chances of misunderstanding

Well of course removing the items from the paper which gave rise to misunderstanding could be beneficial. These items could arise either due to lack of explanation, positioning of clarification, or overselling the results. Lack of explanation happens because we fail to realize as authors that our mind is pre-conditioned while researching on the paper and the reviewer's mind won't be pre-conditioned in the same way. Therefore things which look clear to us may be confusing to a reviewer. Positioning of clarification is very important because not every paper is read word to word. So it is very important to put the clarification or a pointer to it as close as possible to the place where confusion could potentially arise. Overselling does not improve the chances of a paper getting accepted. Overselling of results typically puts the reviewer in a defensive position. A reviewer could look at other existing papers that have introduced similar techniques and be at a loss for what is new, unique, and real about what this paper promises.

So how do you address these problems? One thing is to prepare the paper early and seek feedback. Do not expect somebody, who is not genuinely interested in your work, to provide you good quality feedback for free. You would need to pay. How? Offer the same high quality service on their papers as you expect on your own papers. Posting your papers online, e.g., as a technical report in some archive could also bring some early readership, which may provide you feedback and opportunities to exchange feedback.

If you really need to sell your paper, what's the best way? Give talks -- as many as possible. Try to accept every invitation and try to get yourself invited by marketing the results. In order to market the paper be open to discussing your results in small chats without pen and paper, e.g., over a lunch table. Acknowledge all pre-publication discussions, including those which were not explicitly used in the paper. Mentioning the names of the people is very important, and in case of explicit usefulness, mentioning it explicitly is equally important too. This is so that your colleagues feel acknowledged and positively reinforced to collaborate with you in the future. In the short term, these colleagues are also likely to see the papers more positively vs the case if they find their assistance is not fully acknowledged.

What else can you do if you do not yet have enough opportunities to talk about your paper? We have not done so, but there are cheap as well as free software using which we can easily make a high quality screencast. For me personally, a high quality screencast provides 80% of the benefit of watching the talk in person. Much of the benefit of the remaining 20% can also be obtained if there is an open forum associated with the screencast to ask questions which can either be answered by the authors or other viewers in a relatively short time. Readers do not have the patience unless they are genuinely interested in your result. And expect to count the number of the latter on your fingers.:)

An insider's view of FOCS:

What's specific about paper reviewing these days? As part of the FOCS committee we had access to reviews submitted by the previous STOC committee. We paid a great deal of attention to whether the version we had had responded to the STOC PC's reasons of rejecting the papers. Similarly expect STOC 2008 PC to have FOCS 2007 PC's reviews available. The intersection between STOC 2008 PC and FOCS 2007 PC is non-empty. Even if you think FOCS PC misunderstood your paper, and responding to those misunderstandings would make the paper less readable, you should still try to respond to those misunderstandings instead of ignoring them. In such cases you can respond to those misunderstandings either in appropriate footnotes or in a one page appendix in the end. If your footnotes and appendix are just for STOC PC, do mention "for the reviewers only, will be removed from the published version."

What about the feedback that FOCS PC kept confidential and did not transmit to the authors? This part of the feedback must not be used by STOC committee for three reasons. First, it was understood that only FOCS PC share that feedback. Second, this part of the feedback was a part of the process and not the net outcome. The net outcome ideally must be included in the "send to author" part of the feedback. Third, since this part of the feedback was not transmitted to the authors, they can't be expected to respond. If this part of the feedback did contain a reason why the paper should be rejected then authors must be sent the reason. If this was not done in some cases, then STOC PC must work hard to rediscover the same reason for rejection.

This is my view from both having submitted (and received rejections) papers as well as been part of various PCs. I hope these give you additional practical tical tips on how to best position your papers. I welcome other ideas so that we could continue to improve the quality of our submissions.

Thanks,

Kamal Jain.

Note: FOCS means FOCS 2007. STOC means STOC 2008. Previous STOC means STOC = 2007.

17 comments:

  1. Kamal, good one and thanks for the suggestions. and what do you mean by "overselling"?

    ReplyDelete
  2. Overselling is not an objective term. Subjectively, the authors tend to see more potential of their paper than a reviewer would see. Sometimes this difference in perspective result in the perception of overselling.

    In the real world, people think if they ask $X they would get at least $0.5 X. Therefore if they need $Y they actually ask $2Y. In the papers, this strategy sometimes backfire. I have reviewed papers which claim the possible motivations/benefits from A to Z, thinking that I as a reviewer may at least agree to A to M. I think the paper would do better if the authors mention the most clear cut motivations/benefits and mention some of the most promising ones from their other list. The rest can be move to some other non-important part of the paper or can be kept for an extended version of the paper.

    ReplyDelete
  3. I would pick the following example from Scott Aaronson as one synonym for overselling as I understand it:

    "As far as I can tell, the main impact [of NKS] is that people now sometimes use the adjective 'Wolframian' to describe breathtaking claims for the trivial or well-known."

    More generally, (usually well-meaning) authors may at times use adjectives as in "fundamental problem", "practical algorithm", "straightforward proof" with subjective meanings that reviewers (or readers) might not find agree with.

    As I understand it's not exclusively a problem in academia; also shows up in retail, recruiting, party promoting, etc, etc.

    ReplyDelete
  4. I feel like this is a bizarre post. If your papers are getting rejected from STOC and FOCS so much that you need a strategy for resubmission, then please stop submitting so much and wasting the committee's time (and, apparently, multiple committees' time).

    If more than 50% of your conference submissions are getting rejected, it seems to me that you are employing the following cheap and rather unethical strategy: Knowing that sometimes PCs make mistakes, you are trying as hard as possible to be one of them.

    Here's a question everyone should ask themselves before submitting: Are you submitting a paper that you yourself, were you on the committee, would not accept!? If yes, why???

    (You != Kamal)

    ReplyDelete
  5. Wow, I completely disagree with anonymous #4.

    Papers often get rejected because the authors didn't give the committee enough info to help them appreciate the result; it is quite possible that once the authors rewrite the paper, it will make a huge difference, especially if the feedback sent to the authors is clear as to what needs to be addressed. A rejection of a given version of a paper by a given committee is just that and not the final judgment of the worth of the research, and one should never think of it as such!

    It is especially hard to get a paper accepted if it is one of your first results. It takes a long time to learn to convey technical ideas in writing, you need years of practice. So if this is your first or second paper and it gets in the first time you submitted, think of it as a miracle! And if, as is more likely, it does not get in, PLEASE don't get discouraged, but just use the feedback to improve your paper!

    Yes, someone has to spend time reading those submissions, but hey, one day you'll serve on those committees and pay your dues, so don't worry about that. People serving on program committees volunteered for this job and they understand that new researchers don't always write the most polished papers. Trust me, they will be quite OK with reading your paper even if they know that it was rejected from the previous conference, and they will only think it part of their job to help new researchers improve their papers.

    So: Please just ignore anonymous #4.

    ReplyDelete
  6. Anonymous number 5 has more truth than anonymous number 4.

    To anonymous number 4, try finding a major prize winners who has not gotten his/her papers rejected. You and only you may be surprised to know that their papers also get rejected even after they have won the major prizes. As part of various PCs, I have seen papers by the leaders in their field getting rejected. The only thing a big name guarantees is that their paper will get a serious attention. A reviewer would give a serious consideration to any misunderstanding he/she happen to see.

    Unfortunately, not all papers get equal attention. A big part of the reason is the way they are written. If our paper is not extremely clear, a reviewer may not slog through hours and hours to find the deep hidden meaning of our statements in the paper. That's why it is extremely important to write crisp and clear message. (I by no means implying that I do that myself. In this posting I in fact am trying to figure out why my view differs from the reviewer's view.)

    Accepting or rejecting papers is not a science. This FOCS PC choose the set of papers which were presented. I am sure another PC would choose a different set of papers with a 2/3 overlap. There was a bunch of papers which any PC would accept. There was a bunch of papers which any PC would reject. In the middle, it is basically the preference of this PC. I can say it in the form of the following theorem.

    For every paper which remains undecided after the first day of the PC meeting, there exist a PC which would accept the paper and there also exist a PC which would reject the paper.

    ReplyDelete
  7. Sure, Anonymous #4 is overstating things, but there's some truth to it. Some authors submit only papers they genuinely believe ought to be accepted. Other authors submit lots of marginal papers and hope for the best. This can be very irritating.

    I don't think there's anything wrong with a 25% rejection rate, but if 75% of your papers are being rejected, then you are abusing the system.

    I agree with Anonymous #4 that you should not submit a paper unless you would probably push for accepting a similar paper if you were on the program committee and your arch-rival submitted it. This is not a perfect test, since most people are biased towards their own areas, but if you fail the test, then there's a big problem

    ReplyDelete
  8. ..and instead you may choose to submit to an European conference such as ICALP (dl Feb 10) or CSR (dl Dec 9). Although whp the PC intersection will be non-empty in this case as well.

    ReplyDelete
  9. Sure, Anonymous #4 is overstating things, but there's some truth to it.

    Actually, I don't think I overstated much. Rather, Kamal and anonymous #5 ignored what I said, and chose to respond to an imaginary post.

    And if, as is more likely, it does not get in, PLEASE don't get discouraged, but just use the feedback to improve your paper!

    Of course young researchers shouldn't get discouraged. But this is not the situation to which I was referring. I was (pretty obviously) talking about people who consistently submit marginal papers over and over trying to game the system. (Yes, I agree that if a grad student submits one paper and gets it rejected, they are at 0%. This was not the context of my post.)

    And don't think that committees (well, the good ones) aren't cognizant of who the young researchers are. Often committees are forgiving to strong results which are written and/or motivated poorly by students.

    To anonymous number 4, try finding a major prize winners who has not gotten his/her papers rejected.

    This response is silly. I never said that, once rejected, you should start mowing lawns. Everyone who submits enough to FOCS and STOC will eventually get a rejection. Yes, every (old enough) well-known person in the field has had a rejected paper. But I am quite sure that NONE of them are having > 50% of their FOCS/STOC submissions rejected.

    ReplyDelete
  10. I think one reason the acceptance rates are high in the theoretical CS is that people self select. So we already have a good culture of following anonymous number 7. Of course there may be a small number of abusers. I do not think this blog post is meant to discuss this insignificant numbers of abusers. This blog post is intended to be positively looking. In that regards, I have seen papers resubmitted by our leading prize winners too.

    I do not think anonymous number 4 wrote what anonymous number 7 interpreted. Anonymous number 4 is not writing about one paper which has gotten rejected more than 50% of the time. The statement does not even make sense. Anonymous number 4 is talking about the person. Sort of if more than 50% of your papers are rejected then anonymous number 4 is asking you to give hope.

    The truth is papers which are obvious rejects get rejected early on. Most papers which were not rejected before the PC meeting had some merit. The scientific community would be disadvantaged if those papers are not submitted. Most of those papers will actually get published. Some in the STOC. Some in SODA and ICALP. And some in various other conferences.

    ReplyDelete
  11. When I was writing my comment, I did not see anonymous number 4 had already re-written. Otherwise my response would have been: no response. I do not want to be in any argument with an anoymous.

    ReplyDelete
  12. This has probably been brought up before, but could this blog please copy Schneier's blog and have a note near the comments that says "Real names aren't required, but please give us something to call you. Conversations among several people called "Anonymous" get too confusing." (or at least people follow this by themselves?)

    ReplyDelete
  13. I do not want to be in any argument with an anonymous.

    It's OK, it seems that you are arguing with someone else anyway.

    Sort of if more than 50% of your papers are rejected then anonymous number 4 is asking you to give hope.

    Actually, I said:

    This response is silly. I never said that, once rejected, you should start mowing lawns.

    I'm not suggesting that people "give up hope," just that they aspire to raise their acceptance rates, at least as much as they aspire to raise their total number of accepted papers.

    ReplyDelete
  14. What is the average age of authors with papers in FOCS/STOC?

    I'm curious as to how people manage as they age and their brains deteriorate -- at least in the raw processing power sense.

    Do they coauthor with younger people (e.g., their PhD students)? The combination of experience from an older brain and raw brain power from a younger one may work out well.

    ReplyDelete
  15. If your FOCS/STOC submissions get rejected
    about 50% of times
    at the first attempt, then your papers are better than average submissions ( FOCS/STOC acceptance rate is 25%).
    There is nothing to be
    ashamed of. In fact,
    it would be quite good
    if you are not from MIT
    or Berkeley. But if one paper get rejected many times, that is a bad sign for the paper and you
    should stop submitting
    that paper.

    ReplyDelete
  16. Also one should keep in mind that besides conferences there are also journals.

    ReplyDelete
  17. This X% heuristic for deciding the quality of a paper seems quite arbitrary. The assumption that "gamers" want to get in the paper using only this type of gambling technique may not be true. Every author would like to get their paper in using some legitimate method. If the author does not realize why the paper is getting rejected after trying N times, and the committee does not "get" the value of the paper after N times, then there is a some miscommunication here. It is not reasonable to quickly conclude that the submitter is a "system abuser," IMHO.

    -anonymous mike

    ReplyDelete