(Guest post by
Kamal Jain
Resubmitting the rejected papers from FOCS to STOC?
If your paper was rejected by FOCS and you're submitting it to STOC, here
are my thoughts on how you can increase your chances of acceptance. Given the
low acceptance rate for FOCS, I am sure many of us will be resubmitting
our rejected papers to STOC. Many of us will be incorporating the FOCS PC
comments. And there's also a realistic chance that FOCS PC misunderstood our
papers. So what should we do so that STOC committee does not repeat the
misunderstanding?
Let me first describe the general methods to contain the chances of
misunderstanding. I will then describe why the chance of misunderstanding
has increased for STOC PC on resubmitted papers by giving you an insider view
of FOCS 2007 PC. We can then discuss what we could do to minimize that.
Contain the chances of misunderstanding
Well of course removing the items from the paper which gave rise to misunderstanding
could be beneficial. These items could arise either due to lack of explanation,
positioning of clarification, or overselling the results. Lack of explanation
happens because we fail to realize as authors that our mind is pre-conditioned
while researching on the paper and the reviewer's mind won't be pre-conditioned
in the same way. Therefore things which look clear to us may be confusing to a
reviewer. Positioning of clarification is very important because not every paper
is read word to word. So it is very important to put the clarification or a pointer
to it as close as possible to the place where confusion could potentially arise.
Overselling does not improve the chances of a paper getting accepted.
Overselling of results typically puts the reviewer in a defensive position.
A reviewer could look at other existing papers that have introduced similar
techniques and be at a loss for what is new, unique, and real about what this paper promises.
So how do you address these problems? One thing is to prepare the paper early
and seek feedback. Do not expect somebody, who is not genuinely interested in
your work, to provide you good quality feedback for free. You would need to pay.
How? Offer the same high quality service on their papers as you
expect on your own papers. Posting your papers online, e.g., as a technical
report in some archive could also bring some early readership, which may
provide you feedback and opportunities to exchange feedback.
If you really need to sell your paper, what's the best way? Give talks --
as many as possible. Try to accept every invitation and try to get yourself
invited by marketing the results. In order to market the paper be open to
discussing your results in small chats without pen and paper, e.g., over a
lunch table. Acknowledge all pre-publication discussions, including those which
were not explicitly used in the paper. Mentioning the names of the people
is very important, and in case of explicit usefulness, mentioning it
explicitly is equally important too. This is so that your colleagues feel
acknowledged and positively reinforced to collaborate with you in the future. In
the short term, these colleagues are also likely to see the papers more
positively vs the case if they find their assistance is not fully acknowledged.
What else can you do if you do not yet have enough opportunities to talk about
your paper? We have not done so, but there are cheap as well as free software
using which we can easily make a high quality screencast. For me personally,
a high quality screencast provides 80% of the benefit of watching
the talk in person. Much of the benefit of the remaining 20% can also be
obtained if there is an open forum associated with the screencast to ask questions
which can either be answered by the authors or other viewers in a
relatively short time. Readers do not have the patience unless they are genuinely
interested in your result. And expect to count the number of the latter
on your fingers.:)
An insider's view of FOCS:
What's specific about paper reviewing these days? As part of the FOCS committee
we had access to reviews submitted by the previous STOC committee. We
paid a great deal of attention to whether the version we had had responded
to the STOC PC's reasons of rejecting the papers. Similarly expect STOC 2008
PC to have FOCS 2007 PC's reviews available. The intersection between STOC 2008 PC
and FOCS 2007 PC is non-empty. Even if you think FOCS PC misunderstood
your paper, and responding to those misunderstandings would make the
paper less readable, you should still try to respond to those misunderstandings
instead of ignoring them. In such cases you can respond to those
misunderstandings either in appropriate footnotes or in a one page appendix in
the end. If your footnotes and appendix are just for STOC PC, do mention "for
the reviewers only, will be removed from the published version."
What about the feedback that FOCS PC kept confidential and did not transmit
to the authors? This part of the feedback must not be used by STOC committee
for three reasons. First, it was understood that only FOCS PC share that
feedback. Second, this part of the feedback was a part of the process and
not the net outcome. The net outcome ideally must be included in the "send
to author" part of the feedback. Third, since this part of the feedback was
not transmitted to the authors, they can't be expected to respond. If this
part of the feedback did contain a reason why the paper should be rejected
then authors must be sent the reason. If this was not done in some cases,
then STOC PC must work hard to rediscover the same reason for rejection.
This is my view from both having submitted (and received rejections) papers
as well as been part of various PCs. I hope these give you additional practical
tical tips on how to best position your papers. I welcome other ideas so that
we could continue to improve the quality of our submissions.
Thanks,
Kamal Jain.
Note: FOCS means FOCS 2007. STOC means STOC 2008. Previous STOC means STOC =
2007.
Kamal, good one and thanks for the suggestions. and what do you mean by "overselling"?
ReplyDeleteOverselling is not an objective term. Subjectively, the authors tend to see more potential of their paper than a reviewer would see. Sometimes this difference in perspective result in the perception of overselling.
ReplyDeleteIn the real world, people think if they ask $X they would get at least $0.5 X. Therefore if they need $Y they actually ask $2Y. In the papers, this strategy sometimes backfire. I have reviewed papers which claim the possible motivations/benefits from A to Z, thinking that I as a reviewer may at least agree to A to M. I think the paper would do better if the authors mention the most clear cut motivations/benefits and mention some of the most promising ones from their other list. The rest can be move to some other non-important part of the paper or can be kept for an extended version of the paper.
I would pick the following example from Scott Aaronson as one synonym for overselling as I understand it:
ReplyDelete"As far as I can tell, the main impact [of NKS] is that people now sometimes use the adjective 'Wolframian' to describe breathtaking claims for the trivial or well-known."
More generally, (usually well-meaning) authors may at times use adjectives as in "fundamental problem", "practical algorithm", "straightforward proof" with subjective meanings that reviewers (or readers) might not find agree with.
As I understand it's not exclusively a problem in academia; also shows up in retail, recruiting, party promoting, etc, etc.
I feel like this is a bizarre post. If your papers are getting rejected from STOC and FOCS so much that you need a strategy for resubmission, then please stop submitting so much and wasting the committee's time (and, apparently, multiple committees' time).
ReplyDeleteIf more than 50% of your conference submissions are getting rejected, it seems to me that you are employing the following cheap and rather unethical strategy: Knowing that sometimes PCs make mistakes, you are trying as hard as possible to be one of them.
Here's a question everyone should ask themselves before submitting: Are you submitting a paper that you yourself, were you on the committee, would not accept!? If yes, why???
(You != Kamal)
Wow, I completely disagree with anonymous #4.
ReplyDeletePapers often get rejected because the authors didn't give the committee enough info to help them appreciate the result; it is quite possible that once the authors rewrite the paper, it will make a huge difference, especially if the feedback sent to the authors is clear as to what needs to be addressed. A rejection of a given version of a paper by a given committee is just that and not the final judgment of the worth of the research, and one should never think of it as such!
It is especially hard to get a paper accepted if it is one of your first results. It takes a long time to learn to convey technical ideas in writing, you need years of practice. So if this is your first or second paper and it gets in the first time you submitted, think of it as a miracle! And if, as is more likely, it does not get in, PLEASE don't get discouraged, but just use the feedback to improve your paper!
Yes, someone has to spend time reading those submissions, but hey, one day you'll serve on those committees and pay your dues, so don't worry about that. People serving on program committees volunteered for this job and they understand that new researchers don't always write the most polished papers. Trust me, they will be quite OK with reading your paper even if they know that it was rejected from the previous conference, and they will only think it part of their job to help new researchers improve their papers.
So: Please just ignore anonymous #4.
Anonymous number 5 has more truth than anonymous number 4.
ReplyDeleteTo anonymous number 4, try finding a major prize winners who has not gotten his/her papers rejected. You and only you may be surprised to know that their papers also get rejected even after they have won the major prizes. As part of various PCs, I have seen papers by the leaders in their field getting rejected. The only thing a big name guarantees is that their paper will get a serious attention. A reviewer would give a serious consideration to any misunderstanding he/she happen to see.
Unfortunately, not all papers get equal attention. A big part of the reason is the way they are written. If our paper is not extremely clear, a reviewer may not slog through hours and hours to find the deep hidden meaning of our statements in the paper. That's why it is extremely important to write crisp and clear message. (I by no means implying that I do that myself. In this posting I in fact am trying to figure out why my view differs from the reviewer's view.)
Accepting or rejecting papers is not a science. This FOCS PC choose the set of papers which were presented. I am sure another PC would choose a different set of papers with a 2/3 overlap. There was a bunch of papers which any PC would accept. There was a bunch of papers which any PC would reject. In the middle, it is basically the preference of this PC. I can say it in the form of the following theorem.
For every paper which remains undecided after the first day of the PC meeting, there exist a PC which would accept the paper and there also exist a PC which would reject the paper.
Sure, Anonymous #4 is overstating things, but there's some truth to it. Some authors submit only papers they genuinely believe ought to be accepted. Other authors submit lots of marginal papers and hope for the best. This can be very irritating.
ReplyDeleteI don't think there's anything wrong with a 25% rejection rate, but if 75% of your papers are being rejected, then you are abusing the system.
I agree with Anonymous #4 that you should not submit a paper unless you would probably push for accepting a similar paper if you were on the program committee and your arch-rival submitted it. This is not a perfect test, since most people are biased towards their own areas, but if you fail the test, then there's a big problem
..and instead you may choose to submit to an European conference such as ICALP (dl Feb 10) or CSR (dl Dec 9). Although whp the PC intersection will be non-empty in this case as well.
ReplyDeleteSure, Anonymous #4 is overstating things, but there's some truth to it.
ReplyDeleteActually, I don't think I overstated much. Rather, Kamal and anonymous #5 ignored what I said, and chose to respond to an imaginary post.
And if, as is more likely, it does not get in, PLEASE don't get discouraged, but just use the feedback to improve your paper!
Of course young researchers shouldn't get discouraged. But this is not the situation to which I was referring. I was (pretty obviously) talking about people who consistently submit marginal papers over and over trying to game the system. (Yes, I agree that if a grad student submits one paper and gets it rejected, they are at 0%. This was not the context of my post.)
And don't think that committees (well, the good ones) aren't cognizant of who the young researchers are. Often committees are forgiving to strong results which are written and/or motivated poorly by students.
To anonymous number 4, try finding a major prize winners who has not gotten his/her papers rejected.
This response is silly. I never said that, once rejected, you should start mowing lawns. Everyone who submits enough to FOCS and STOC will eventually get a rejection. Yes, every (old enough) well-known person in the field has had a rejected paper. But I am quite sure that NONE of them are having > 50% of their FOCS/STOC submissions rejected.
I think one reason the acceptance rates are high in the theoretical CS is that people self select. So we already have a good culture of following anonymous number 7. Of course there may be a small number of abusers. I do not think this blog post is meant to discuss this insignificant numbers of abusers. This blog post is intended to be positively looking. In that regards, I have seen papers resubmitted by our leading prize winners too.
ReplyDeleteI do not think anonymous number 4 wrote what anonymous number 7 interpreted. Anonymous number 4 is not writing about one paper which has gotten rejected more than 50% of the time. The statement does not even make sense. Anonymous number 4 is talking about the person. Sort of if more than 50% of your papers are rejected then anonymous number 4 is asking you to give hope.
The truth is papers which are obvious rejects get rejected early on. Most papers which were not rejected before the PC meeting had some merit. The scientific community would be disadvantaged if those papers are not submitted. Most of those papers will actually get published. Some in the STOC. Some in SODA and ICALP. And some in various other conferences.
When I was writing my comment, I did not see anonymous number 4 had already re-written. Otherwise my response would have been: no response. I do not want to be in any argument with an anoymous.
ReplyDeleteThis has probably been brought up before, but could this blog please copy Schneier's blog and have a note near the comments that says "Real names aren't required, but please give us something to call you. Conversations among several people called "Anonymous" get too confusing." (or at least people follow this by themselves?)
ReplyDeleteI do not want to be in any argument with an anonymous.
ReplyDeleteIt's OK, it seems that you are arguing with someone else anyway.
Sort of if more than 50% of your papers are rejected then anonymous number 4 is asking you to give hope.
Actually, I said:
This response is silly. I never said that, once rejected, you should start mowing lawns.
I'm not suggesting that people "give up hope," just that they aspire to raise their acceptance rates, at least as much as they aspire to raise their total number of accepted papers.
What is the average age of authors with papers in FOCS/STOC?
ReplyDeleteI'm curious as to how people manage as they age and their brains deteriorate -- at least in the raw processing power sense.
Do they coauthor with younger people (e.g., their PhD students)? The combination of experience from an older brain and raw brain power from a younger one may work out well.
If your FOCS/STOC submissions get rejected
ReplyDeleteabout 50% of times
at the first attempt, then your papers are better than average submissions ( FOCS/STOC acceptance rate is 25%).
There is nothing to be
ashamed of. In fact,
it would be quite good
if you are not from MIT
or Berkeley. But if one paper get rejected many times, that is a bad sign for the paper and you
should stop submitting
that paper.
Also one should keep in mind that besides conferences there are also journals.
ReplyDeleteThis X% heuristic for deciding the quality of a paper seems quite arbitrary. The assumption that "gamers" want to get in the paper using only this type of gambling technique may not be true. Every author would like to get their paper in using some legitimate method. If the author does not realize why the paper is getting rejected after trying N times, and the committee does not "get" the value of the paper after N times, then there is a some miscommunication here. It is not reasonable to quickly conclude that the submitter is a "system abuser," IMHO.
ReplyDelete-anonymous mike