The processes of submitting FOCS/STOC abstracts and conducting PC meetings have undergone numerous changes since the good old days when you received your acceptance letter by US Mail and a couple of weeks later you received a huge rolled-up bunch of poster-sized papers on which you were supposed to glue your paper and mail back. There is little doubt that these changes have improved efficiency and fairness a great deal.
I would like to propose another, somewhat more radical, change that is now technologically feasible -- allowing people to submit, together with their 10 page abstract, a 10 (or 20?) minute video describing their result. The video will be optional, at least in the beginning.
They say a picture is worth a thousand words -- if so, a 10 minute video is worth millions! Imagine, as a PC member, how much easier it will be to read an abstract after you see a short video explaining the problem, the approach, and the main new ideas, and how much more "correct" your evaluation of the paper would be! In my opinion, this will greatly improve quality of the paper acceptance process. Many people complain that the latter is currently broken -- a large fraction of the decisions are nothing more than the flip of a coin or are left to such chance events as who reviews the paper or the constitution of the PC.
Many objections can be raised to this idea. Let me anticipate a couple and try to counter them. First, this change is feasible today -- if you need proof, just take a look at YouTube! Another objection is that this may give an advantage to some members of TCS community -- those who can give better talks. But then, they are precisely the people who are also better at writing clearly and already had a huge advantage. In fact, in my opinion, relatively speaking, the enhanced process will be a great equalizer -- giving a chance to people who don't have good writing skills to still be able to sell their wares.
Needless to say, this is a major change and it deserves an extensive discussion before it is implemented. I hope this blog will provide that opportunity.
Interesting, but I think this is a terrible idea for many reasons.
ReplyDeleteFirst, although uploading a video may not be a problem, I would have no idea how to go about producing the video. (I don't consider the videos I can make on my cell phone or digital camera of sufficiently high quality.) Wouldn't this have the effect of shutting out people who don't have access to fancy video equipment?
Second, I don't see what the benefit is as compared to a well written paper. The only advantage of a presentation is the potential for interactivity (ability for the reviewer to ask questions) but that does not seem to be present in your approach. I would much prefer giving authors the chance to send (written) responses to PC questions rather than having to produce a video up front.
Finally, it seems to me that the overhead of producing a video in addition to a paper will be unacceptably high.
Vijay mentioned the reduced burden on the PC, the possibility of better decisions (both worthwhile goals) and the feasibility of doing this but didn't emphasize the increased burden on the authors. How soon before the deadline should one start on the video? All those authors who spend the last few minutes before the deadline beefing up their intros and making them coherent will now be encouraged to (eventually have to) design videos to go along with them.
ReplyDeleteHow would it change things?
A video is currently required for SIGGRAPH submissions; tweaking the details of those videos occupies much more time than writing the papers. I would not want to trade our current style of deadline stress for theirs! How can we get the benefits without the cost?
I think this idea also gives unfair advantage to people who can prepare "good" (entertaining?) videos. Yes this advantage is already present for people who can write well, but I think it is easier to learn to write well than to learn to speak well. (At least that is my experience.)
ReplyDeleteHere are some replies:
ReplyDeleteFirst, I believe a simple webcam can produce a reasonable video.
Second, you are under deadline pressure - would you spend the time to add a couple of figures in your abstract which will make it much easier for the PC to understand your ideas? I believe a good video will decrease the pressure on you to write a sparkling writeup. Moreover, the former is much easier to pull off than the latter. Imagine how much easier it is to go to the board and show your proof to some colleagues than to actually write it up!
This is a good idea, but a rather radical one that may require a change of mindset before it can be adopted.
ReplyDeleteThat said, the negative aspect would be that there will be *added* pressure. Given how computer scientists operate, there will be pressure not only to produce a dazzling writeup, but also a dazzling video.
I am afraid that a pretty girl will then have more chances for a positive review than an old crook. At least, from myself :)
ReplyDeleteWhat if they find out that she is a dumb blond?
ReplyDeleteConjecture: such a thing will never happen.
ReplyDeleteif the problems are the quality of the video or the appearence of the submitters then a mp3 file would be a mildier approach.
ReplyDeleteBut what about author anonymity?
ReplyDeleteThere is a major tradeoff here which (I think) no one has yet explicitly pointed out.
ReplyDeleteThe main benefit of this would be to have video abstracts of most/every paper. If videos are easier to understand for the committee, they are easier to understand for the entire community. This could be a great asset. I am not entirely sure why there aren’t more videos of presentations out there already (like of every conference talk at STOC/FOCS).
Thoughts?
The main draw back would likely be less polished papers. If submitters are under a time crunch at the end and must make a video, they will likely put less time into their paper especially the abstract and introduction as these are the areas likely to be covered in the video. This could be a disaster, or could be much ado about nothing.
Thoughts?
Certainly an interesting idea. One worrying thing as someone pointed out earlier is that people might take the videos too seriously (both the authors and referees). As long as it is clear it is for ease of understanding and not to judge the quality of the paper, the method might be useful. Otherwise, we have the SIGRAPH model which I believe no one wants to incorporate where people put more effort in the video and less in the paper.
ReplyDeleteYou don't even need a webcam, just a microphone.
ReplyDeleteIt's easy to make a video of what appears on your computer screen, together with audio, so you could just prepare a short presentation and record that. No one needs to see what you look like. And if you have slides already made, it really doesn't take very long. Here are some examples of screencasts. Here's some free Windows software for making a screencast.
Ben Toner.
The links in the post above don't work. These ones do:
ReplyDeleteHere are some examples of screencasts. Here's some free Windows software for making a screencast.
Ben Toner.
Similar points have been raised above, but I will make it more radical: In my opinion going the video route would decrease the average correctness of the accepted papers. Even if the committee would be able to maintain their balance (they wouldn't), the writers will have to concentrate on their rhetoric (and in TCS terms 10 minutes can contain nothing but rhetoric) instead of spending the time to make a more coherent extended abstract. Having to write coherently helps people to catch their own mistakes, making a hand-waving speech does not. And it may be that we are stretching the QA done in the correctness front a bit thin already.
ReplyDeleteI believe a good video will decrease the pressure on you to write a sparkling writeup.
ReplyDeleteIt is hard to believe that you think that this would be a good thing.
I'd probably screencast a slide presentation, as Ben points out. Or, mix in some whiteboard. Still, I'd rather not have the extra burden. I like that people feel pressured to make their papers well-written, since it makes it easier to read them. This would take authors' attention away from the writing with no benefit to me (except when I'm reviewing).
ReplyDeleteAs an author myself, I'd rather spend my time polishing the writing, because there are large economies of scale. Every hour I spend improving my paper's presentation will help all the thousands (millions?) of people who read that paper. It's so worth it! :) On the other hand, I'd probably spend several hours preparing this video, a few people would see it, and then the video would be of no further use to anyone.
I don't think the video abstract is a good idea. First of all, we should keep in mind what is the ultimate goal of the whole publication process: select the best contributions and present them in a clear and serviceable form for the fellow researches. All changes should be made to get nearer to this ultimate goal.
ReplyDeleteNow, the video goes in the opposite direction: it introduces (willingly or not) "marketing" factors in the judgment process that hamper, rather than ameliorating, the objectivity of the review. Surely, some of these factors are also present in pure writing. But I think that we should go in the direction of decreasing their influence, not increasing it.
I cannot see videos becoming mandatory with STOC/FOCS submissions, but nothing prevents anybody at this point to make a video and give the URL link together with their submission.
ReplyDeleteBeyond videos, there is the possibility of putting up a powerpoint presentation, which may help people present the paper in the future (students for example read and present STOC/FOCS papers often in their reading groups; an existing powerpoint presentation may help a lot...)
In fact, since nearly everybody gives powerpoint presentations in their FOCS/STOC talks, we may start collecting them and putting these
presentations out anyway.
I don't like the idea.
ReplyDeleteAs an author, I don't want the extra burden, with little to no reward.
As a referee, I'd prefer to be able to look at the paper and go straight to the section I'm interested in (videos are too linear for this kind of browsing). In fact, most often when I have to review a paper, I skip most of the "marketing" and go straight to the section that defines the model and states the results.
Since nobody has challenged my statement that the FOCS/STOC paper acceptance process is currently broken, should we assume this is common knowledge?
ReplyDeleteIf so, what should be done to repair this quintessential TOC institution?
what should be done to repair this quintessential TOC institution?
ReplyDeleteHaven't you seen this?
Since nobody has challenged my statement that the FOCS/STOC paper acceptance process is currently broken, should we assume this is common knowledge?
ReplyDeleteIt would seem so. The database community has taken a proactive approach to their submission process and they are actively testing improvements to the system.
By the way, have a look at the September 2006 SIGMOD article on the occurrence of bias in non-anonymous refereeing.
URL for SIGMOD article is:
ReplyDeletehttp://acm.org/sigmod/record/issues/0609/p06-article-tung.pdf
"The Myth of the Double-Blind Review?
ReplyDeleteAuthor Identification Using Only Citations"
http://www.acm.org/sigs/sigkdd/explorations/issue5-2/Task4-Place3.pdf
Please spare us a rehash of the same old discussion about program committees!! At least almost everyone has focused on the novel idea in the post rather than being dragged down into the morass.
ReplyDeleteLooking forward, Milena Mihail's suggestion of capturing presentations could be great. Many workshops ask for these after the fact since they can't expect any of them to be done ahead of time. To deal with the variety of incompatible formats that are all being displayed on the same data projector the simplest idea may be some sort of digital capture off the data projector itself. It would have the advantage of not adding any overhead for speakers.
To expand on Milena Mihail's suggestion, I think we should capture videos of all conference talks and put them online along with the presentations.
ReplyDeleteThis can be very powerful: for instance, the software that is used by Microsoft to put talks online allows you to correlate the video with the slideshow, and you can navigate based on either the video or the slides. Listening to a talk in this manner is almost better than being there, because it lets you go at your own pace.
Anyway I think that's something that's much less controversial and easier to implement, but nevertheless very beneficial.
"The Myth of the Double-Blind Review? Author Identification Using Only Citations"
ReplyDeleteSo in the worst case the author is identified and we are back where we are right now, in the best case, the guess is all wrong, and the refereeing ends up being for all purposes anonymous.
Can you explain in which way is this worse than what we have now?
Also observe that the hit ratio is 50% for a system that is actively trying to identify the authors. In practice most reviewers wouldn't waste time trying to guess when the probability of being right is, at best, 50%.
Regarding hit rates for guessing authors, I have some experience with this (from years of reviewing Crypto submissions). In practice it's really easy to figure out the authors, especially considering that many papers are posted on the internet. I always try, mainly because not being told bugs me. Probably many reviewers don't but I'd guess most do.
ReplyDeleteThe situation is even worse for anonymity than one might guess from the statistics. Most of the difficult authors to identify are pretty obscure. Certain forms of discrimination become harder (against women or ethnic minorities), but I doubt these factors play much of a role. By far the most prevalent form of discrimination is in favor of people who are already well known or are at prestigious institutions. That form of discrimination isn't stopped by double-blind refereeing, since being unidentifiable is a strong sign of lack of fame.
To anon 29:
ReplyDeleteMy point still stands: In the worst case the author is identified, in the best case, the guess is all wrong, and the refereeing ends up being anonymous.
This is still an improvement over the current system which is always not anonymous.
Furthermore, the figures from SIGMOD suggest going blind had a modest, but measurable positive impact. So why oppose it then?
Accepting papers from more famous people is a reasonable thing to do as such papers are more likely to be correct.
ReplyDeleteGiving money to the rich is a reasonable thing to do because the rich are more likely to use money wisely.
ReplyDeleteThere are several issues here:
ReplyDelete- Discrimination: Videos will make it easier to identify reputable members of the community. They also may introduce biases based on ethnicity, sex or other issues (this is also true for mp3's, to a lesser extent). Regarding this issues, videos can only be harmful.
- Paper "value": Will videos create a situation where better papers are accepted? This can go either way. On the one hand, the committee is choosing someone to give a talk at a conference - the level of video the person can produce is an indicator of how well that person will talk. An article may be amazing but unsuitable in some way for oral presentation, and then it should appear in a journal and not in a conference. On the other hand (as mentioned above), a talk can more easily "mask" errors. Here there are advantages and disadvantages.
- Good video vs. good paper: Obviously, having both a good paper and a good video is best. I'd generally be more willing to spend 10 minutes understanding the general ideas of a paper through a video or presentation, than I would be to read the paper, but others seem to feel otherwise. Separating the deadlines for papers and for videos will reduce the trade-off in quality issues - have the videos due 2 weeks after the paper...
Gilad.
Well, I will challenge the main statement: The FOCS/STOC submission and acceptance system is not broken. At least, not broken enough to merit a major reform. The rules are clear (no high point of entry), simple (one can concentrate on content rather than procedure), and acceptance is just correlated enough with genuine paper quality to make "prove an interesting result" the dominant strategy. What more can you ask for?
ReplyDeleteThe video suggestion is very bad.
ReplyDeleteHere are some (partial) points I have:
1. If a video presentation is better than a paper, then why don't start publishing videos instead of papers?
2. If you think that conference acceptance mechanism is problematic, why don't stop supporting and organizing TOC conferences; and move to a more serious mechanism of journal submissions?
3. Writing has an important ontological status in mathematics. Switching to web presentations has no scientific advantage.
to make "prove an interesting result" the dominant strategy
ReplyDeleteThis is not so. The dominant strategy is "prove a complicated result" preferably complexity related.
Don't believe it? We had a solution to a long standing open problem rejected from STOC/FOCS because "the proofs were not particularly complicated".
For a successful experiment in archiving some fabulous talk for posterity, see:
ReplyDeletehttp://www.cs.caltech.edu/%7Eschulman/cs-lens-1.html
http://www.cs.caltech.edu/~schulman/Workshops/CS-Lens-2/cs-lens-2.html
The cost was modest - less than the travel expenses of 2 participants! (This is perhaps the best venue for informing the TOC community of this amazing repository.)
Archiving FOCS/STOC talks is a no brainer. Imagine the next time you have to delve deep into a difficult paper and you have access to the author's 20 minute talk!
I have a slightly different take on Milena's suggestion. I think the current presentation system is broken.
ReplyDelete1. There is no feedback system for talks. Many times the speaker does not realize that only if they spoke slower / tried to present less material / used bigger font / ... their talk would be much better and much more accessible.
2. On the flip side, there is very little motivation for the speakers to make a good presentation. Perhaps if we were to record the presentations this would be a step in the right direction.
I don't understand!?
ReplyDeleteIf everyone would have his/her talk recorder, why should there be a conference committee at all? Or even a conference at all?
Everybody can simply watch the presentation of the paper he/she likes.
Yeah. Why have conferences at all? Why don't we all just post videos of our talks and use a reputation system to determine what's worth looking at. No messing with program committees, or airplanes, or hotels. Less expense and less wear and tear on the environment. No need to argue at business meetings about whether we should have printed proceedings or just CDs. We can all simply sit in our offices or homes and watch videos instead.
ReplyDeleteAs it is, we can just sit in our offices and shop for books, CDs ...
ReplyDeleteThis is not so. The dominant strategy is "prove a complicated result" preferably complexity related.
ReplyDeleteDon't believe it? We had a solution to a long standing open problem rejected from STOC/FOCS because "the proofs were not particularly complicated".
One shouldn't read the comments
from the pc too literally. One
person's "long standing open
problem" is another person's
"not so interesting, nobody
has really thought about it
open problem".
And some times
papers get rejected because
there is a limit to the number
that can be taken - some thing
has to give.
Anonymous #6:
ReplyDelete"I am afraid that a pretty girl will then have more chances for a positive review than an old crook. At least, from myself :)"
Weren't we trying to increase the number (and prominence) of women in TCS anyway? :)
I am very surprised how much opposition there is to video, particularly since this discussion is taking place on a blog (where I assume the sample bias skews the other way).
ReplyDeleteA video is no substitute for a well written paper, but a paper isn't generally a substitute for an informative talk. A video, if done well, is.
Almost everyone sees value in attending talks. Why isn't some of that value obtained by watching videos? Video submissions could even be posted online with the option to comment. This would allow viewers to ask questions or even discuss the results.
Making a video is not hard, even without a webcam. Just take some digital photos of your whiteboard, record audio as you draw/write, and make a slideshow using any number of programs.
We all had to learn latex for papers. Learning to make a slideshow is much, much easier. With a little practice, I doubt it would take more than a few hours. To avoide last minute panic, the videos could be due 24 hours after the paper.
The readers of this blog may well get more out of talks than videos, but what about researchers in other fields? advanced undergraduates? people around the globe who can't attend FOCS/STOC? Future members of the theoretical computer science community?
Video is a promising medium. Perhaps vijay and others would be willing to make a sample submission and post it on this very blog to see what people think.
a paper isn't generally a substitute for an informative talk
ReplyDeleteA paper should be fully informative to a professional.
The whole video idea sounds more like another step back:
First TOC went from respectable full journal papers into two-columns short sketches of proofs presented in conferences.
Now people suggest that we should accompany our short sketchy papers by a promotional video clip.
What's the next step?
To sum it up:
Why not go into journal-based community like other respectable fields?
One person's "long standing open problem" is another person's "not so interesting, nobody has really thought about it open problem".
ReplyDeleteThis is a red herring. If the paper is not at STOC level because the question is not very interesting then the referees can say so. Instead they wrote "proof is not difficult enough".
First TOC went from respectable full journal papers into two-columns short sketches of proofs presented in conferences. Now people suggest that we should accompany our short sketchy papers by a promotional video clip.
ReplyDeleteWhat's the next step?
CS moved to the conference-as-primary-dissemination-medium due to its fast pace of development. This made sense back then and arguably still makes sense today. The problem was that (i) the community decided not to allow double counting of conference and journal versions thus removing the incentive to publish the full version in a journal and (2) keeping STOC/FOCS small hence making it absurdly difficult to publish there. To give an idea FOCS'80 had 420 pages, FOCS'06 had 748, STOC'80 had 446 pages STOC'06 had 766, with most of the growth taking place in the 80s, viz. the number of combined acceptances is roughly the same as fifteen years ago. Meanwhile the number of worlwide TCSers grew by an order of magnitude.
One person's "long standing open problem" is another person's "not so interesting, nobody has really thought about it open problem".
ReplyDeleteThis is a red herring. If the paper is not at STOC level because the question is not very interesting then the referees can say so. Instead they wrote "proof is not difficult enough".
You seem to be concluding that the system is broken from one example. This is like arguing that all graphs are planar by showing the example of K_4.
I agree that there are bad decisions. I also agree that there are good papers that would repeatedly get rejected from FOCS/STOC because the proofs are simple and not obfuscated. The system has problems, but to call it broken is a stretch.
Also, I don't think videos will help solve the above problems.
This is anon 36&46
ReplyDeleteThe system has problems, but to call it broken is a stretch.
I didn't offer the example as proof that the system is broken. It was offered as evidence that STOC/FOCS pay far too much attention to difficulty over quality. In our particular case we introduced a completely new technique and yes, using this novel technique the proof is rather simple. A mathematician would describe this as elegant, a STOC/FOCS reviewer in contrast calls it trivial and rejects the paper.
I do not think videos would help either, and neither do I think the current STOC/FOCS state of affairs is a disaster, though it could certainly use a tune up. Essentially we are running the same system as 15 years ago, even though TCS grew and changed substantially in that period of time.
Very cool Vijay!
ReplyDeleteLance, I do not have to send you (in fact Bill Gasarch now). To others, Lance agreed to give me some space for asking why do not we use web 2.0 tools in collaboration.
Vijay's point is not exactly the same but some of the comments are. I think we are stuck in web 1.0 which was good for speeding up communication.
Some examples: Go to http://www.researchchannel.com. It has hundreds of video from Microsoft and other institution. If you ever gave a talk at Microsoft it might be there. Look at the UI. I do not see why Stoc/Focs talk can't be presented there.
Another example with more web 2.0, go to http://channel9.msdn.com/ Here you will see that people could even ask questions on the video. Either the author or other co-watchers could answer the questions.
Eventually I think Vijay is on a bit wrong track here. I think Stoc/Focs have lived there life in their current format. If we want we can continue them. But they are losing importance way too fast. There was a time when stoc/focs paper were good enough to get you a job. But not any more. If we see in our group here, there is not much correlation between stoc/focs papers and the candidates we choose. Sometimes applicants with 10 stoc/focs papers lose to applicants with one or sometimes even 0 stoc/focs papers.
We should let the scientific process evolve. And one such evolution is not to depend upon the reputation system as provided by the stoc/focs, with 3 over worked experts.
Instead put your thoughts/papers/talks on the web 2.0 (not only web 1.0). Get the opinion not only from 20 experts but possibly 200 experts. I personally have a lot of inertia but still I do particpate on at least 5 to 10 topics each week. I believe that's what going to happen, web 2.0 will empower the right set of people with the right set of ideas.
There were a lot of typos in the above. But I am not going to correct them because the meaning is still unambiguous. Instead I like to add some more information, hopefully without as many typos.
ReplyDeleteOne reason I could not deliver on my promised post to Lance on using web2.0 is that I wanted to implement my proposals as a proof. Otherwise I look hypocrite. I wanted to create a screencast of my own talks and putting them in web 2.0 space before suggesting others to do. If I do not follow my own proposal then that’s a proof that my proposal is not that easy. That's also a proof that all it is good for is telling others to do:)
Having said that, I have been in discussion with Uri Feige and many others. In particular I am asking Uri’s permission to put all the accepted STOC papers/talks (with author’s permission of course) in web 2.0 space. This is done as an experiment for seeding purpose and we could have sought the resources from Microsoft to do so. The plan got stuck because we both agreed that the process should evolve. And If somebody likes to experiment with the plan, then it is better that the experiment originates in a university. The goal should be to increase and improve the scientific production due to better collaboration.
"Making a video is not hard, even without a webcam. Just take some digital photos of your whiteboard, record audio as you draw/write, and make a slideshow using any number of programs."
ReplyDeleteIt's not that easy. Whiteboard pictures are hard. There's always glare, there are shadows. Many flaws aren't noticeable in person, but look awful on a slide.
I think if the videos are publicly available, it puts a lot of pressure on doing a very good job. Therefore, it will take a very long time. If the videos are only available to reviewers, that would be a lot easier. I don't really want to spend time advocating my paper to a tiny group of people instead of improving the paper for everyone -- but understanding that the referees are putting in a substantial amount of their own time, I don't think it is too much to ask.
Here is an example of how such a video could look like:
ReplyDeletehttp://www.bbc.co.uk/berkshire/features/divide_zero_sum.ram
This one was recorded by BBC.
CS moved to the conference-as-primary-dissemination-medium due to its fast pace of development.
ReplyDeleteConferences, and especially ridiculously competetive ones, do not help in the fast dissemination of results. On the contrary, they hold back things and damage TOC (waste of money, waste of time polishing conference papers to have good chance of acceptance, hiding of recent sent results, publishing mistaken results due to partial proofs, etc.).
Journals can do a much better job to help the field.
Conferences, and especially ridiculously competetive ones, do not help in the fast dissemination of results. On the contrary, they hold back things and damage TOC
ReplyDeleteThey do, but this is not unique to TOC. Most (all?) other areas also have these extremely competitive premier conferences in which often just having a high quality result is not enough and second order non-quality considerations end up being the deciding factor.
It would certainly be a good start if we can record talks in FOCS/STOC.
ReplyDeleteI have no idea about this but I believe that this might be expensive as hotels might charge large fees for it. Does anybody know whether such a thing was thought about before and why was not it implemented.
Journals can do a much better job to help the field.
ReplyDeleteThe average delay between my journal submissions and final publication has been around 2 years. Journals are less freely available than proceedings are and have typically made more demands that keep papers off author web pages. Even with a year of conference rejections, papers would be disseminated faster through conferences.
Even with a year of conference rejections, papers would be disseminated faster through conferences.
ReplyDeleteThis begs the question.
If people were not investing most of their energies on conferences, then journal publications might be faster.
Further, if the delay in journal publications is the prime factor here, why not try to improve this, instead of suggesting things like "video clips for talks", etc.?
A/V costs are a shockingly high proportion (say 10%) of the cost of running a conference. It is not just the equipment (though hotels won't let you use their sound systems etc unless you run things through their offices.) It is also the cost of staff to be on site to handle issues when things go wrong (which, with laptops changing every 20/25 minutes, is bound to happen). University A/V departments are not much cheaper, BTW.
ReplyDeleteIn the controlled environment of a small room one can almost entirely automate the video production process, as Kamal is used to at Microsoft Research. Automation is the only way that I can imagine something being cost effective. It isn't so clear how easily one can do this in a large room with indifferent lighting like many of the ballrooms in which conference talks are held. The only way this could work is if it did not involve a camera.
My one other concern is that to record talks we would also need each presenter to sign a release form or somehow otherwise explicitly give their permission to do this which would be another painful item to heap on the conference organizers.
I am really surprised by the number of people objecting to videos..
ReplyDeleteFirstly, videos are no alternative for papers(conference or journal). The videos only aid the reader. I do not see anything wrong in making use of the technology available today to have more effective dissemination of ideas.
One can have the video deadline a week after the paper deadline. That way, authors can concentrate on the video after the paper is completely ready.
Admitted, it is easy to handwave proofs in a video. But the real proofs will be written in the paper anyway.
I havent been in the conference committee anytime.. but my guess is that for any given paper only a few of the committe members have read the paper. With a video talk, it is possible for the entire committee to get a quick overview of the paper. I think this can only aid in making better decisions.
Even if sending videos to program committee is not agreeable, at least the actual conference talks should be videographed and put up on the web.
Journals have their own advantages in terms of correctness of proofs etc. But conference system is not as broken as it is made out to be..
It really does not matter if other fields(math) value journals more, we do not have to ape them.
If people were not investing most of their energies on conferences, then journal publications might be faster.
ReplyDeleteFurther, if the delay in journal publications is the prime factor here, why not try to improve this, instead of suggesting things like "video clips for talks", etc.?
You aren't going to change 40 years of ingrained habits. You also haven't addressed my other point that many of the organizations running journals have been bad actors w.r.t. easy dissemination of results. (They have different interests from researchers in the field.) So... in addition to replacing conferences by journals you would need to replace most of the organizations producing them as well. Good luck getting this to work!
There are plenty of papers in STOC/FOCS with rather easy proofs. The theorem statement or the proof has to be above certain threshold and then there is the issue of competition and the taste of the particular PC. The system is not perfect for sure but perhaps the following summarizes the issue faced by the PC (borrowed
ReplyDeletefrom Suresh Venkat's blog).
http://www.paulgraham.com/judgement.html
The theorem statement or the proof has to be above certain threshold and then there is the issue of competition and the taste of the particular PC.
ReplyDeleteHere's that canard again. If the statement was uninteresting they could have easily said so. If you have served in PCs you know by now that reviewers are quite candid about their opinions given the blind nature of the review process.
Oh for Pete's sakes, polish the write-up and send the thing to another conference. Either SODA/CCC/ICALP/SOCG/whatever or the next STOC/FOCS that comes up. They can't have dingbats on the committee every time.
ReplyDeleteEverybody gets a raw deal once in a while, but if your result is an elegant solution to a longstanding open problem then surely another PC will appreciate this.
---STFUAGBTW
Paul, I agree that good videos are difficult and expensive. We have a whole team working on this. Even then they typically need 24 hours notice to be able to record a talk.
ReplyDeleteIn the business world it is a popular saying is that ideas are dime a dozen, it is the execution stupid (the usage of "stupid" as in KISS). Youtube is no proof here for easyness, the videos on youtube are quite different. All youtube is proving is that this a widely considered idea. We need to ask two questions ourselves.
1. Why it is not already happening?
2. If it is really that easy, then why did not Vijay first created sample videos on his own paper and then write this post with links to the sample videos as proof?
The answer to both the questions are related. It is not happening because either our scientific world has not yet seen the value and/or the because of our inherent lazy nature, the amount of work needed is non-trivial and/or there is not enough motivation. It easy to say than actually do.
I do not personally think STOC/FOCS should make video as a requirement. As Milena suggested people who wants to create video can always create the videos and include the links in their submissions. If video is really helpful then it may increase the chance of acceptance.
Flickr and digital photography predates videos by a long margin. If picture is thousand word, then people could always have added several pictures in the end as an appendix or a link to PPT or a picture folder. If people are not already doing thousand words, why would people do millions? Certainly a video could be an appendix. In fact one could embed a link in PDF. In case the reviewer is reviewing the paper online, he/she does not even have to type the link but simply click it.
So video should be a personal choice and I welcome people to create one. I am in FOCS committee, and I promise to watch the videos of the papers assigned to me, even though looking at the appendix is an optional thing.
The other issue Vijay is pointing is about the STOC/FOCS review process. Vijay is claiming that STOC/FOCS review process is broken. I agree to some extent but I do not think it is only the process itself. It is sometimes the authors. Let me take a concrete example.
I submitted three papers this upcoming STOC. The least worthy of them got accepted. The one on which even the authors have disagreement. I think the fault lies with the authors of the papers and pretty much any other review process with video or no-video would have resulted in the same outcome. We as authors of the accepted paper included a lot of marketing junk in the paper. There was so much marketing junk that it was bordering lies. The statements like the algorithms we are proposing are already proposed 30 years ago were included but not highlighted. Why we authors choose to put a lot of marketing junk? The answer is that we knew that our reason why this paper is important is not going to convince the committee members who would want to see some "new" technical details and not only "new" philosophical details aka "perspective". When "perspective", which we think is very important, is the main value of the paper then it becomes a hard sell.
As Mohammad said above, the low hanging fruit to improve the STOC/FOCS papers is to cut the marketing junk off. Many of us submit essentially "work in progress" instead of "completed work". Why can't we wait for the next conference?
There have been various suggestions I have heard to improve this process. One of them is to epublish all the submissions just like archiving. We in industry could be disadvantaged a bit in terms of patentability, but I am personally willing to take that little disadvantage.
Another was to epublish all the accepted submissions at least from the time the results were announced till the time conference happens. This will also speed up the scientific discovery process by speeding up the spread of the knowledge.
These are some methods of public accountability. If your submission is going to be public, it is likely to be cleaner for the reviewers too.
Let us pluck the low hanging fruits and leave the rest on the evolution. If people like they could include links to the videos in upcoming FOCS submissions as an appendix. If there is not enough time, post the links to your webpages later. There is a chance that a reviewer may watch the video. Esepcially if, as claimed, that video could simplify and improve the job of reviewing.
Oh for Pete's sakes, polish the write-up and send the thing to another conference.
ReplyDeleteOh, it already appeared in another good conference a while ago. I don't dwell on this in the least bit, it just so happened that it was relevant to a comment made by Eldar about quality in STOC/FOCS.
The responses were, however, quite interesting in themselves. Two people argued that it really didn't happen (the referees were speaking in tongues), and about yours there is little to say after that colorful signature.
The whole point of this video thing is lost on me. Isn't it just a way of asking for longer abstracts in disguise?
ReplyDeleteAdam
Instead of requiring videos, people could instead share their reading notes to enhance their insight of research papers.
ReplyDeleteI've been working on such a service:
http://studystickies.com
Kamal,
ReplyDeleteI am surprised by your admission
regarding your stoc submission that
got accepted. Why do people of
the stature of Vijay and yourself
need to get down to the level of
overselling your work? How do
you expect a pc constrained by
time to make reasonable decisions
while authors do all they can to
obfuscate the true merit of their
paper?
{\em The whole point of this video thing is lost on me. Isn't it just a way of asking for longer abstracts in disguise?}
ReplyDeleteAs I am writing up a paper for FOCS, I find this page limit of 10 extremely frustrating. After much economizing, I find it very hard to fit all the ideas inside the page limit. I'd hate to sacrifice the quality of the writeup in order to conform to the limit.
My understand is that the page limit is to make the job of the referee easier. Going by this logic, a video would definitely make the referee's job easier while allowing me to communicate my ideas better.
You aren't going to change 40 years of ingrained habits.
ReplyDeleteIsn't this exactly what Vijay tries to do in this post?
As I am writing up a paper for FOCS, I find this page limit of 10 extremely frustrating.
ReplyDeleteStrictly enforcing the 10-page limit (for submissions rather than final versions) does not make much sense. The general guideline makes a lot of sense: write the paper in a single column easy-to-read format; make the most important part easily accessible and 'up front'. However, 10 pages of double-column format fits a lot more than 10 pages of single column format.
Anonymous@69:
ReplyDeleteThis is a very good question but irrelavant to present discussion.
If you like to know the answer, drop me a separate email. (First name followed by the last initial @ microsoft.com).
The anecdote was relevant as an example. You think others do not oversell their papers? I think others also do but I rather not pick examples from others. Some put marketing junk more than others.
The anecdote shows that the conferences have become some sort of markets instead of a forum for scientific communication.
I think the current interactive nature of web (which is being termed web 2.0) provides revolutionary methods of scientific communication. These methods need to evolve themselves rather than forced as Vijay seems to be conveying here. The motivation must come from masses.
Communication->collaboration->communication is a virtuous cycle. Web tools can really make this virtuous cycle fast. If anybody wants to research/implement some of the methods, then I will be more than happy to collaborate on such tools. Again my email is the first name followed by the last initial at microsoft.com.
What is the line between a thorough comparison with pre-existing research and "marketing junk"?
ReplyDeleteWhat is the line between describing a technique and "marketing junk"?
Anonymous@74. Great questions!
ReplyDeleteA thorough comparison brings out literature which is supportive of the work as well as the literature which is not that supportive of the work.
A thorough comparison credit the past work properly, even if done by another community. It does make clear what insight was exactly missing and what is provided. It also puts the compromises made upfront.
A thorough comparison uses language which highlights the scientific truth. If something is objective then say it objectively. If something is a subjective opinion, then make sure that the personal opinion is not portrayed as a widely believed opinion. For an example, this problem is believed to be the holy grail of computer science.
Think of it, if it was not your own theorem then how would you be writing about it? For an example in a book. It is hard but that's the gold standard.
A well-written paper is always easier to understand than a talk. If you think you've gotten some information out of a 10-minute power point talk/video, you're actually worse off, because you think you understand something you really don't.
ReplyDeleteAnon #76 is right.
ReplyDeleteA (well written) paper is always better than a short talk.
Neil Postman,a well known media theorist, claims that audiovisual communication creates merely the illusion of understanding. Here's the obligatory wikipedia reference:
ReplyDeletePostman asserted that by its very nature, television confounds serious issues with entertainment, demeaning and undermining political discourse by making it less about ideas and more about image. He also argues that television is not an effective way of providing education, as it provides only passive information transfer, rather than the interaction that he believes is necessary to maximize learning.
"A (well written) paper is always better than a short talk."
ReplyDeleteBy this argument, there is no point having talks in conferences, we can just get the proceedings by mail..
Recall, that here a video is *not* touted as a substitute for a paper, but only as an additional resource.
To Commentator @ 79.
ReplyDeleteI love talks and I do find the additional resource useful. But since this seems to be a logical argument I like to contribute a bit more logic here. There is a difference between a public talk given in a conference and a talk supplied to make review fairer.
In public, your talk gives a glimpse of the paper so that the audience can find out whether the paper is relevant/interesting for them to read.
Reviewers are required to read the paper for a fair evaluation. Time always has a zero-sum game. If reviewers are already over worked, then by definition the time to watch the talk will come from the time to read the paper. Therefore "substitution" aspect comes in.
This is anon 76 again.
ReplyDeleteto anon 79: Hopefully talks given during a conference have a different purpose than papers submitted to reviewers. In particular, reviewers should be forming their own interpretation of the results of a paper very quickly.
People attending a conference have the luxury of forming a fuzzy opinion and coming back to the paper later. And talks and videos are perfect for generating interest and forming fuzzy, often incorrect, opinions :)
I find talks to be almost entirely useless. Their only value is that they sometimes help make up for terrible writing. Lots of academic writers have stupid hang-ups. If they explain their ideas too clearly or admit that they originally had even higher goals, they fear looking less impressive. If they include motivational heuristics, they fear that their writing will be too informal. The net effect is that they publish a dense, formal account and then explain things in talks (where the lack of permanence helps them let their guard down). That's better than never explaining anything, I suppose. However, if someone's talks are easier to understand than their writing, it means their writing is incompetent.
ReplyDelete10 page limits really don't help, since they turn adding more explanation and motivation into a zero-sum game. The underlying problem behind all of this is that the paper with the best chance of acceptance is very far from the paper that would have the most value to the community after publication. Some wise people substantially edit their papers after acceptance (to remove exaggerated marketing and add explanation), but it's sad that they have to have the first version at all.
Anon@82, I love your comment. Especially the ending, >>>Some wise people substantially edit their papers after acceptance (to remove exaggerated marketing and add explanation), but it's sad that they have to have the first version at all.<<<
ReplyDeleteThere had been a lot of discussions, I think initiated by many leaders including David Karger, about the process of STOC/FOCS is too slow. It takes almost half a year when the work was submitted to the work when published. There were several suggestions to speed up the process. One of them is never got challenged, but still not implemented.
Publish the accepted submissions at the same time when the list of accept submissions is published.
This will also make authors trying to start writing their papers weeks in advance rather than days in advance.
The only criticism doing proposal had is that it is not achieving the as many goals as other more complicated proposals, like getting rid of paper proceedings, were achieving.
But something is better than nothing. At least publishing the accepted submissions make the communication faster. Since this require almost no additional work, we should start doing this asap, until we are ready to make more revolutionary changes like no paper proceedings. Note that FOCS 2007, already requires to submit the paper electronically.
Excellent idea. First, let us see what a theory person does - works out concepts, proves things AND THEN writes this up and sends it to conference. Now what is the big deal in him explaining the work he has already done. In fact the video need not even contain sketches of proofs.
ReplyDelete1. Bias - As if there is no bias today. Instead of stopping this good idea it would be better to propose ways to solve the bias problem. In fact, with videos, a group of people can simultaneoulsy do the review. Here is a theory problem in itself. How to compose review groups (from a larger pool) such that bias is minimized/removed.
2. Correctness - Once selected, the presenter has to produce a complete paper (complete means matching what the preson claimed in video) which can then be assigned to one reviewer for correctness check. A correctness failure can then go thro a second round of review with more people etc.
Cost/effort to make videos is not really an issue..