I argue that computer science uses conferences to play the role of reputation that journals play in other fields for reputation but then conferences no longer focus on the more important role of bringing out community together.Our conference system forces researchers to focus too heavily on quick, technical, and safe papers instead of considering broader and newer ideas. Meanwhile, we have devoted much of our time and money to conferences where we can present our research that we can rarely attend conferences and workshops to work and socialize with our colleagues.Computer science has grown to become a mature field where no major university can survive without a strong CS department. It is time for computer science to grow up and publish in a way that represents the major discipline it has become.
You can also download the pre-publication PDF.
Update 8/3: The editors of CACM have made the full text of the CACM article publicly accessible.
8/7: Collection of related blog posts and other links. Feel free to send me others.
Where do you get the following statistic? "For example, nearly half of the Godel prize winners .. were initially rejected or didn't appear at all in the top theoretical computer science conferences." The latter part is easy to verify, but what about the former?
ReplyDeleteAs a comment, if theory does move to journals, I don't think page charges would help people switch.
Good luck on your quest!
Hear, hear. I commend this article.
ReplyDeleteIn addition to your justifications, I would add one of my own. You state, "Most researchers don't have the time and/or money to travel to conferences where they do not have a paper." This is true, but in addition, many researchers do not have the money to travel to conferences where they *do* have a paper. Journals cost nothing to authors to submit or publish. Even if they did, it could not possibly rival the cost of airline ticket + hotel + registration fee. Since I largely fund my own travel out of a meager graduate student stipend, this is has led directly to money-based decisions when I send a paper to one conference over another. For publication venues that determine the prestige of my CV, My only questions should be, "Is this venue appropriate?" and "Is this venue prestigious?"
The current system is like a tax on success that only the rich can afford to pay. Combined with the fact that just to get an academic job interview, a student must graduate with a publication record that, two decades ago, would have merited tenure, implies that this tax must be paid perhaps a double-digit number of times before a student can even earn a Ph.D.
Certainly money begets success (and hence more grant/promotion money) in other, less direct ways, and we should not try to systematically eliminate every such imbalance. But in this one case, we could greatly level the playing field, while correcting all the other problems mentioned in Lance's article. By paying attention only to journal publications, where publication is free, a scientist working on a shoestring budget could rise from obscurity to fame completely through journal publications, like a Horatio Alger story.
I had never seen that 1999 CRA report. One wonders if Drs. Patterson, Synder, and Ullman, who wrote this:
ReplyDeletethe more complete level of review (4-5 evaluations per paper compared to 2-3 for an archival journal)
have ever received a conference review. Saying that conference reviews constitute a "more complete level of review" because sometimes there are 4-5 of them is like saying that water adds more flavor to my soup than salt or garlic because I put more water in it.
I concur by and large, Lance. (And, I admit to being lax in journalizing some of my papers.) I will repeat a comment I made on Noam Nisan's blog: I see no reason why our journals can't guarantee fast turnaround times -- say, <= 3 months for the initial report unless the paper is very long -- "Science" and "Nature" are very fast, for example. Since CS papers these days have maybe more than 3 authors on the average, it suffices for the average researcher to referee about twice the number of papers they write. Fast turnaround at journals will be essential if we want journals to regain their position. Perhaps we could have the related discussion on this blog about good writing practice for CS that will make refereeing and reading a pleasure -- something that Terry Tao has done on his blog for math writing.
ReplyDeleteTheory of Computing is one example of a fast-turnaround journal. (I am saying this unbiased by the fact that I am an editor for this journal.) I think such examples should be emulated.
One of the ways computer science can grow up is by not hiding our most important results---or even enlightening editorials---behind paywalls.
ReplyDeleteIf it's not public, it's not a publication.
I don't quite understand how you expect to increase conference attendance by shifting focus to journals. If researchers are struggling to pay for conferences now when their academic performance depends on it, why do you think they would pay to go to a conference for socializing? Also, as someone already pointed out, a shift towards journals will happen only after the turnaround time becomes measurable with conferences. Waiting for 2 years to get your paper published is just not acceptable for many researchers, and it defeats the purpose of communicating your ideas because the scientific community reads about your research two years after you've done it -- this kind of lag is just not acceptable.
ReplyDeleteIsn't the turnaround time for CS journals slow because many people don't care about journal publications? Why not postpone your journal reviewing if the journal doesn't matter anyway! Journal reviewing will speed up when the prestige shifts from conferences to journals. Senior researchers (who make the hiring decisions) first need to declare that journals, not conferences, are what carry the prestige, and the rest will follow.
ReplyDeleteThe turnaround time is not slow just due to priorities. There's also the fact that conference papers are at most 8 pages, whereas journal papers are usually 30+ pages (often 50+), which takes considerably more time and effort from the reviewer.
ReplyDeleteIt is very nice that this discussion -- conference vs. journal publications -- started at all. It was long due to ask "why we are so different?" Science is done by people. It is nice to meet then and there, to drink a good vine, and discuss not finished yet, even "stupid" ideas with friends an colleagues. Isn't this the main role of conferences?
ReplyDeleteBut current CS conferences turned to something very different -- a place of hard competition. And a quite expensive undertaking. The "human aspect" got lost completely. Even worse: a "half-cooked" result accepted (in a hurry of PC) by a "prestigious" conference is declared as a *result*. (Fortunately, the proofs of many good results accepted there are at least sketched.) What then happens (a journal publication, a rare event) is secondary. Why are we (CS community) so different? Just because our field is so young?
Would it not be better to put things in their normal place (as other fields do): conferences for meeting friends and discussing "hot things", and journals for collecting/archiving the *real* outcome?
There also are other strange things happening in CS. Say, SICOMP (like FOCS/STOC) welcomes "complicated" proofs---too "easy" proofs (those from THE Book) have no chance?
In any case, the questions Moshe and Lance raised are very important. We lose not only founding. We lose young people. They choose fields where "science is still science", not a pure competition. Not mentioning the problems like "who will give me these $1000 ++ to meet my enemies".
this is has led directly to money-based decisions when I send a paper to one conference over another.
ReplyDeleteI do the same thing. As an example, I had a paper accepted to DISC 2008. I did not submit to DISC 2009, even though I "could have," because I could not afford the plane fare.
Journal reviews take so long because the same reviewers also have to handle conference papers.
ReplyDeleteThis of course increases the overall workload, but also I suspect that many reviewers prioritize conference reviews over journal reviews (at least I do).
It's only natural: conference reviews come with a strict deadline whereas a journal review can always wait a little longer, right ?
As they stand now, journals cannot gain more prestige in CS. Journals are all about locking away papers where nobody can access them, and that does not make sense to most of us.
ReplyDelete"Waiting for 2 years to get your paper published is just not acceptable for many researchers, and it defeats the purpose of communicating your ideas because the scientific community reads about your research two years after you've done it -- this kind of lag is just not acceptable."
This is why everybody uses the arXiv. I think that it is not acceptable *not* to post your papers on a permanent online archive---no, your homepage does not count.
"We lose not only founding. We lose young people. They choose fields where "science is still science", not a pure competition. Not mentioning the problems like 'who will give me these $1000 ++ to meet my enemies'."
ReplyDeleteI can't understand several of these sentences. I do not agree with what I do understand. Every field in science is as competitive as computer science. Competition is a big part of science. I think young people enjoy it.
I hope competition does not have to be a big part of science. And, as a young graduate student, I certainly don't enjoy it.
ReplyDeleteIn my department, faculty tell us to be extremely careful about who we share our on-going work with. Except for a few "known friends", we typically don't tell anyone even roughly the problems we are working on until a paper has been submitted. There is a sense among students of who is the biggest competition, inside and outside the department, for the tiny number of top academic positions. It's poisonous!
Every field in science is as competitive as computer science.
ReplyDeleteThis is not true. The extent of competition in CS and especially TCS is bigger than many other fields.
Competition is a big part of science.
This is wrong. Science has nothing to do with competition. Science is meant to be a universal thing, where competition is local and subjective.
I think young people enjoy it.
This is baseless.
Competition is a big part of science. I think young people enjoy it.
ReplyDeleteCompetition suggests a sense of urgency as well as selfishness and animosity. But speed-researching in the mathematical sciences (or any field for that matter) would not lead to deep results.
ReplyDeleteJournals are all about locking away papers where nobody can access them, and that does not make sense to most of us.
On the contrary. Conference proceedings are much harder to get online than journal papers. (Try IEEE and ACM; you'll need a subscription to get their conference papers).
This is why everybody uses the arXiv. I think that it is not acceptable *not* to post your papers on a permanent online archive---no, your homepage does not count.
ArXiv and ECCC are very good, but posting on your own homepage is more important. The authors' homepage is certainly the first place most people nowadays look for a paper.
In the time it takes a program committee to review the papers for a conference, we could referee an entire year of a journal.
ReplyDeleteWhile we are doing all this work for our conferences, we just won't have the time to have everyone's papers appear in journals.
That said, we should never expect to have the quick turnaround times that journals such as Science and Nature achieve. The amount of work required to referee a mathematical paper is at least an order of magnitude larger than that required to referee an experimental paper.
I find that to do it well I need uninterrupted time. That means I do most of my refereeing over the summer or winter break. So, if someone submits a paper in January or September, they are going to wait a few months, even if my queue is empty.
"ArXiv and ECCC are very good, but posting on your own homepage is more important. The authors' homepage is certainly the first place most people nowadays look for a paper."
ReplyDeleteIf true, that is too bad. The community needs to standardize for definitive and permanent online paper versions.
"There is a sense among students of who is the biggest competition, inside and outside the department, for the tiny number of top academic positions. It's poisonous!"
How is this poisonous? It should be motivating. Pretending that there are more than a tiny number of even decent academic positions will not make it so. There are at least ten times as many well-qualified people looking for academic jobs as there are positions. This competition can't be swept under the rug.
"But speed-researching in the mathematical sciences (or any field for that matter) would not lead to deep results."
Independent researchers trying to solve problems often leads to major results faster. Consider the human genome project, the Netflix prize, the DARPA Grand Challenge. In math, consider the three proofs of Poincare's conjecture in higher dimensions. In computer science, see FOCS and STOC.
Do these include "deep" results? I would not argue against it. Of course there are counterexamples, such as Perelman's proof. Science takes all types and is not simple to characterize. And what applies to one field does not apply to others. Theoretical computer science is perhaps deeper than economics, but is less deep than some areas in pure math. I also think that Perelman is a special case.
I agree that Theoretical Computer Science has matured as a mathematical discipline and its publication culture should be brought in consonance with the publication norms in mathematics of our times. This entails:
ReplyDelete1. Only journal articles should count as publications.
2. Conference presentations should be considered for what they are -- invited presentation of results to be published in journals.
3. Reform the conference system, so that they become venues for meeting and exchanging ideas, rather than semi-annual beauty contests.
4. Reward research which cannot be packaged into a 10-page sexy package for STOC/FOCS. This includes developing long term programs for attacking the central problems of the field (around the P/NP question) (analogies in other fields include the Langland's program, Thurston's geometrization conjecture,
the Mori program etc.) Such programs are the hallmarks of a mature discipline and should become the center of gravity of research in TCS as well-- replacing the current culture of "publishing" short 10-page conference papers.
ArXiv and ECCC are very good, but posting on your own homepage is more important. The authors' homepage is certainly the first place most people nowadays look for a paper.
ReplyDeleteTotally agreed. I am a student, and on several occasions I've started looking for a paper in 'official', standard repositories only to end up looking for the author's name in Google and finding the paper in their homepage. When I don't find the paper in the website I write an email to the author requesting it. It has worked for me the three or four times I've tried it, and I even got Dexter Kozen scanning an article for me because it was so old he didn't have a digital copy.
If people are willing to share their knowledge/papers in that way then I don't see a reason not to look in their personal webpages first, especially given that most official repositories require payment to get material from them.
Anon wrote:
ReplyDelete"In my department, faculty tell us to be extremely careful about who we share our on-going work with."
Frankly, this is silly advice. In at least 90% of the time, discussing your partial results widely will allow them to come to fruition much faster (either due to feedback or maybe even with collabortaion). Even if you loose credit 10% of the time, openness is highly advantageous to you personally. And this is even before we take into account all the indirect benefits of living in an open atmosphere.
By definition, there will always be a shortage of the best jobs, grants, and prizes. Again by definition, this will lead to competition for these. The question is how do you participate in this competition. I simply do not believe that in today's system (in TCS, at least) there is any real personal gain to be had from "anti-social" competition.
Where I go first to look for a paper that I know the title or author of is neither arxiv (despite how helpful I find it) nor the author's home page, but Google scholar. Google can do the going-to-arxiv-or-homepage part itself well enough.
ReplyDeleteWhat I find arxiv more useful for (and author homepages far less useful) is giving me a feed of recent papers in the area that I wouldn't have known to go looking for.
"What I find arxiv more useful for (and author homepages far less useful) is giving me a feed of recent papers in the area that I wouldn't have known to go looking for."
ReplyDeleteYes. Perhaps it is because I am in a smaller department, but otherwise I first hear of many papers when they are announced for a conference. "Checking people's homepages" seems to be a method for promoting cliques. How many homepages can one check? Really, only those for a couple people whom you know about, and the vast majority of research, some of which might be fascinating, flies right by.
Maybe a good start to prove that CS is a "maturing" discipline of mathematics would be to catch up with the rest of math in how we distribute preprints.
This will reduce the role of conferences as gatekeepers to awareness and reduce the poisonous competition mentioned above. (I have never received any advice to keep my research secret, but at least the arxiv makes precedence more clear than hidden homepage papers.)
Personal homepages are certainly not "hidden".
ReplyDeletePosting on a public repository and on your homepage is almost the same.
If you are concerned with the lack of automatic aggregator, I would say it's only a tiny technical/software problem to write such a service for personal homepages delivering latest manuscripts uploaded to ones homepage.
People could then subscribe to such a service, and save their own preferences and settings.
I would guess this will be even more popular than ArXiv in some sense.
"In my department, faculty tell us to be extremely careful about who we share our on-going work with."
ReplyDeleteNo offense, but this is the mark of a mediocre department. Being a good, joyful participator in collaboration and conversation about open questions is part of your obligation as a scientist --- and it will add, in the long and even the short term, to your reputation almost as much as your next publication will.
Of course, if you contribute meaningfully to the conversation, the other participants have a moral obligation to give you credit for it, include you as a coauthor, and so on. People occasionally act unethically and steal ideas, but this tends to rebound on them and their reputations very quickly. On balance, being open about your ideas is good for you.
I believe that a fundamental change is called for in the way most TCS people tend to write their papers. Writing a paper should always mean a paper intended to be published in a journal (or sometimes in whats called an "edited proceedings" such as Symp. of Pure Math. or Contemp. Math etc.) The "conference version" of such a paper
ReplyDelete(if needed to be produced at all) should then be a shorter version "abstracted out" of the main paper already in existence.
The current modus operandi of many TCS researchers (particularly graduate students and young researchers) is exactly the opposite. They tend to wait till the last minute and produce hastily written "10 page extended abstracts" to meet a deadline, and often not produce a journal version at all. This process needs to be changed if TCS is to be considered a mature field.
And here is the reason for this "opposite modus operandi", from Lane's paper:
ReplyDelete"By deemphasizing their PUBLICATION ROLE, conferences can once again play their most important role: Bringing the community together."
Just at the point! I only wonder WHO could stop this "madness?" The PC's will definitely NOT do this: just because being in a PC of say FOCS/STOC/CCC is important from a very pragmatic side. Should this not be then done by us, the community? What would happen if we would send to these conferences not our "compressed" 10 pages, but just a 1-2 page abstract of the proposed talk? Just as other (e.g. mathematicians) do?
Journals cost nothing to authors to submit or publish.
ReplyDeleteFirstly, there is a large cost to the community. Journals are not cheap.
Also, I just got an email from Elsevier regarding a journal version. Apparently, I can get my figures printed in color for a price of $720.84.
Frankly, while I may agree with the sentiment of writing papers in journals, I cannot get myself to support publishing in locked expensive journals that use the community to get all their work done for free and then basically charge everyone as much as they can.
PS: I understand that I have the option of getting my figures in b/w instead and that is what I'll do. But I just find the principle distasteful.
I cannot get myself to support publishing in locked expensive journals that use the community to get all their work done for free and then basically charge everyone as much as they can.
ReplyDeleteBut the IEEE and ACM proceedings are also locked and expensive, not to mention the expenses on the conferences themselves.
Also, many new good journals are becoming free and electronic.
By definition, there will always be a shortage of the best jobs, grants, and prizes. Again by definition, this will lead to competition for these. The question is how do you participate in this competition. I simply do not believe that in today's system (in TCS, at least) there is any real personal gain to be had from "anti-social" competition.
ReplyDeleteThis idea of science being a competitive
sports in which scientists are engaged in to win laurels is harmful, childish, and to be honest a little disgusting. That having said -- competition surely exists in the real world. But it should be considered as the dirty underside of our profession which we should publicly decry whenever and wherever we see it, rather than glorify it. Competition is not what drove most great scientists and mathematicians. Think Riemann, Ramanujan, Galois and a host of others.
To anyone thinking of research as a "competitive sport" -- I say just visualize Ramanujan wrting in his final notebooks while dying of TB, and ask yourself -- was he motivated by "competition" ?
Noam Nisan has written a very cogent counter-argument in praise of conferences and trends that discusses advantages of our conference system. He expresses this much better than I can.
ReplyDeleteA key to Noam's argument is the positive role of conference PCs in helping set research directions for the field, keeping young researchers out of ratholes and directing attention to areas of likely and worthwhile progress.
Why couldn't journals do the same? A typical journal paper is read by 2 or 3 reviewers selected by one editor who may in turn have been selected by the author. Though the number of readers (and certainly the depth of reading) on a typical major conference PC may not be greater, the key is that those readers must justify their assessment of the paper to the rest of a group of 20 or more researchers from across the entire field. This level of assessment focuses not just on the narrow technical details of the review but also on how well the work is advancing the field as a whole. It also assesses the relationship of the work in this sub-area to work in other sub-areas and can find new connections of which the authors are unaware.
No journal in CS provides that kind of overall evaluation and potential for direction-setting.
(In the worst case all that one needs is a small clique of mutual admiration, one member of which has managed to become an editor, to perpetuate a narrow subfield working down a rathole.)
There are different properties that we might want of a publication medium:
(1) speed
(2) selection
(3) correctness
(4) availability
(5) longevity
Major conferences win easily on speed and selection relative to journals, but sacrifice some on correctness. Major conferences are at par with society journals and beat for-profit journals on availability.
Internet publication means that conferences no longer are the best for speed (or availability). The arXiv is the clear winner on speed and availability but completely ignores selection or correctness. ECCC sacrifices some speed for a minimal amount of selection but still beats conferences on speed. All of these publication methods beat self-publication on home pages for longevity.
Our current default mode is arXiv/ECCC + conference + journal dissemination. If we cut out the conference part of this then we can improve the speed of journals a little and get the corrected versions faster but we still do not recover the selection advantages that conferences have.
Reward research which cannot be packaged into a 10-page sexy package for STOC/FOCS. This includes developing long term programs for attacking the central problems of the field.
ReplyDeleteIndeed. We've recently embarked in a long term program to solve a top problem in the field. We have made good progress, solving several questions along the way and chipping away significantly at the problem. A complete solution remains years away though.
Problem is conferences tend to dislike the program nature of our work. By "program", I mean: this is where we are going, these are the twenty or so pieces that need to come together, here are reasons why this is a sound strategy, this paper addresses issue 8.
Instead, conferences tend to seek neatly packaged results that fit in ten pages. As a consequence we publish most of our results in journals only, with the inconvenience that in CS people do not read journals for new results anymore.
Moreover our grad students pay the price when applying for a job since they don't have wads of papers in the main conferences.
A key to Noam's argument is the positive role of conference PCs in helping set research directions for the field, keeping young researchers out of ratholes and directing attention to areas of likely and worthwhile progress.
ReplyDeleteI think to be more honest one should say
"keeping young researchers out of HARD PROBLEMS and directing attention to EASIER newly invented areas of likely and worthwhile progress." -- which is a questionable goal when one thinks in terms of the interests of the subject, rather than in the interests of the young researchers.
I read Noam's post and I found that it emphasizes the role of short-term competition as the major driving force of TCS. As if science is driven by a Wall Street type process in which PC members (mostly young researchers these days) performing the "almost mythically heroic" role of the traders on the floor. This model is aeons away from how more mature fields (like mathematics, physics etc.) operate. Lets not get too much carried away by this make-believe world of Wall Street voo-doo economics -- a look at our respective 401K's should convince us if not anything else.
It is very amazing that we (the CS community) are perhaps the ONLY who seriously discuss the role of journal publications. People from other fields take it as natural. We have (unfortunately) not so many alternatives as yet. All are not happy with publishers (these "fat guys" making money from our job)---but there are new happenings like "Theory of Computing." This bolsters up.
ReplyDeleteBut I think that Lance's question was not "what to do with journals", but rather "what to do with our CONFERENCES?" How to re-make them to be meetings of friends, of ideas and not just a "battle place" of "hiring points."
Noam argument falls on its face when we see the algorithmic game theory work in CS. Except some complexity results, most of it is shallow in comparison to economics literature. Many stoc/focs papers are simple corollaries of economics papers. So as long as we are evaluating within our community, it is fine. But if we talk about science, we are behind. Algorithmic game theory is the only branch of science I know, which is studied in two different fields, one serious and other not that much.
ReplyDeleteAre there other sub-areas in computer science which are studied in two different areas. We can see if TCS wins or the other. So far TCS score is 0 out of 1.
TCS has developed algorithmic game theorists at no where close to what economics have. If economics noble prize is extended to algorithmic game theorists, we do not have even a single one who could even be a footnote. This is irony, since the current need of economics requires CS expertise. If TCS is a serios science, the at least in algorithmic game theory we should be work which will stake claim on economics noble prize in a decade or two. You will be surprised that economists will learn CS expertise and win over so CS economists or algorithic game theorists.
Somebody said, you get what you measure. We are bean counters, so we get a lot of beans. Otherwise I do not know, how can somebody write 3 serious papers in a year, when other disciplines requires one serious paper in 3 years? The question is, is it because we are super smart, or is it we produces beans which let us fart later.
It is very amazing that we (the CS community) are perhaps the ONLY who seriously discuss the role of journal publications. People from other fields take it as natural.
ReplyDeleteI do not think this in itself constitutes an argument. Other more mature fields do lots of other things (e.g. grad students' and postdocs' lives are much harder; they are treated as labor, advisor is on every paper, much more politics, etc). I don't believe we would want to adopt those things.
In general, convincing-101 will tell us that no argument is helped by being condescending, calling the other side childish/immature, and pretending that your side of the argument is the self-evident truth.
I however do agree that overall, it would be better if our conferences accepted twice as many papers and were (resultingly) less competitive. I still don't see a place for closed, private and expensive journals.
(In the worst case all that one needs is a small clique of mutual admiration, one member of which has managed to become an editor, to perpetuate a narrow subfield working down a rathole.)
Think string theory.
In general, any system has its tradeoffs. It is easy to see the things that are wrong with the current system. Much harder to see things that are right, and would be wrong in the alternate proposal.
Are there other sub-areas in computer science which are studied in two different areas. We can see if TCS wins or the other. So far TCS score is 0 out of 1.
ReplyDeleteWhat about quantum information science studied by physicists as well ? Do physics
papers in this area cite TCS work in a meaningful way ?
"Other more mature fields do lots of other things (e.g. grad students' and postdocs' lives are much harder; they are treated as labor, advisor is on every paper, much more politics, etc). I don't believe we would want to adopt those things."
ReplyDeleteMath does not put the advisers name on each student paper. But they have journal publications and non-competitive conferences. No one is saying that reform means exploitation of students.
I completely agree with this, specially after the last congress I went to, I got so bord with all the talks, and I remember I used to love going and learning stuff. We need to really be able to innovate, not just to publish those damn paper to acomplish the expectation of the career.
ReplyDeleteJust to throw in another angle on how bad "conferences" can be. A conference was organized (not any time recently) in honor of the 60th Birthday celebration of a senior professor. Let us call him S. I did not go to the celebration but I talked to a brutally honest senior professor who attended the celebration. He was disgusted that they kept calling S a researcher. He said S did a lot of service work for the complexity theory conference during its nascent years, which is why S rose to prominence.
ReplyDeleteLet us look at this objectively: S did a lot of service work for a conference, rose to prominence, and then we have a conference
to celebrate his 60th birthday and call him a researcher!
You can argue that the birthday celebration was a pity party: S did a lot of thankless service work, so what is the harm in making an old man happy on his 60th birthday. That argument works till you take into account that S used his prominence to destroy the tenure of a brilliant young professor. A young professor whose work would have advanced the field, unlike S's work which advanced only S's career. This story is not based on hearsay; I know all the parties involved first hand.
" I still don't see a place for closed, private and expensive journals."
ReplyDeleteI am not sure commercial journals offer the only alternative. For example, many of the best journals in math (Journal of the AMS, Annals of Math ...) are published by professional societies, and hence (relatively) cheap. Also, why are they "closed"? Anyone can submit a paper there.
Situation in Quantum Information System is not as bad as Algorithmic game theory. I think we could claim 1/3 point there.
ReplyDeleteIndeed some aspects such as Shor algorithm is purely CS. But if we look quantum at the physical layer, we are again far behind. It is not that we do not publish paper in Quantum at the physical layer. But most of our papers in Quantum seems to be accepted at lower "percentile" level than other CS branches.
So the score is 0.3/2 or 1/6. 0/3 in algorithmic game theory, and 1/3 in quantum. Look at the recent algorithmic game theory book. Three out of the four editors of the books have published rediscoveries paper demonstrating their lack of knowledge of the game theory in general. If you talk to them in person, you would quickly realize that often they do not know the well known stuff in game-theory, oh but algorithmic game-theory is different. But they can't objectively define how different, except by saying, what we know is algorithmic game theory.
Our top conference in algorithmic game thoery, EC makes arbitrary decisions. Our top CS conferences such as stoc/focs makes even more arbitrary decision in algorithmic game theory.
If the people in algorithmic game theory are smart, could they offer a solution to the following puzzle so that the the PC members of the top conferences are incentivized to make the right decisions, i.e., not serving the PC, or resigning is at higher priority that doing the bad job at reviewing. Accountability is the other word. Can the community make PC accountable for a bad job or reward them for a good job?
To finish the comment on a positive note, I feel indebted to Christos and Costis. They at least save our face, by studying the problems which we are good at. Their work is of the top most quality.
Major conferences win easily on speed and selection relative to journals
ReplyDeleteThe speed point seems to be spinnable either way.
one direction: journals have slower turnaround time, so authors have time to meditate on their results during review, and find broader implications of their techniques and better/simpler methods of presentation, before the final paper appears. The paper then appears in a form that has greater impact potential. Getting the results known in 24 hours can always be done simultaneously via arXiv.
the other direction: conferences have faster turnaround time, so results get out faster, and the community can then process these results and progress faster. A more enlightening writeup can always be submitted to a journal later.
"Three out of the four editors of the books have published rediscoveries paper demonstrating their lack of knowledge of the game theory in general."
ReplyDeleteI believe one such result being refered to is the paper by Kleinberg and Tardos about bargaining which appeared in STOC 2008. What are the other results? Would help to educate other AGT people.
Also, this also happens in other areas, for example, optimization, e.g. paper in FOCS 04 of Charikar and Wirth in which they reproved a known result using the same algorithm.
In other areas, we have a fair share of noteworthy work too. In AGT, we can't even see ourselves in mirror. If our community book carried the name of Christos, it would be honorable to all of us. Not only he has done some great work, he figured out what we as TCS experts could contribute too.
ReplyDeleteTardos Klienberg paper and a followup appearing in upcoming focs is a embarassment not because they are rediscoveries but because they are well known results. I do not want to point out myself, but if you know game-theory you would know 2 out of the other 3 are in the same boat.
Their papers are published not because of the results they achieved, but because of the unnecessary technical error-prone complexity they have introduced in relatively straighforward to prove theorems. Because our PC members gets the credit for just, being on a PC and instead of appreaciating the scientific accomplishement of a paper to humans, they could measure the paper based on the technicalites they can't understand. If they can understand a proof, it must be too easy. If you have been in a PC, you could read a review like, "I have not verified the proof, but they look technically sophisticate so I like to recommend accept", nobody says, "I have not verified the proof, but they look needlessly complicated, and the authors have not paid effort to simplify them. So I like to recommend to return the paper back to authors for reviews".
Lance paper he linked to is also not backed by the data. What is the percentage of declines, when a person is invinted to sit on STOC/FOCS PC? If it is low, then it means, either the work load is not too much, or the credit is too much, or the expectation is low, or the cost is low. In practice it is all of the above.
I should also add that, there are third types of reviews too. "I like the theorems, but the proofs are too simple for Stoc/focs conference" but it is hard to see a review, "The paper is a borderline by the theorems, but the simplicity of the proofs make it an accept".
ReplyDeleteThough it is a simple game-theory, that if understanding is a proof is easier then the proof would be applied more broadly and the science expands. Scientists who really won't submit the paper, unless they can derive a proof intutively are punished. In economics, it is appreciated. That's the difference in culture, why you get shallow results in algorithmic game theory as done in TCS. Attend an economics talk, and you will see.
"I do not want to point out myself, but if you know game-theory you would know 2 out of the other 3 are in the same boat."
ReplyDeleteThis is a ridiculous statement. Why will the "embarrassing followup" appear in FOCS 09? Because no one publicized that the stoc 2008 result was not new. So now, we will have followups on the other "known" results.
On the one hand you accuse these people of not knowing something that "everyone in econ" knows, but when prompted to find out what it is that they in econ all know, you will not tell. It does not help.
Some of the work on cost-sharing and market eq by Vazirani and some of the work on congestion tolls by Roughgarden.
ReplyDeleteLet me put the table otherway round. Point out a fair share of truly great work by them or should I say us? Tardos and Roughgarden have done some remarkable work on Price of Anarchy. This is sort of computer sciency and sits in the same category as Christos work (though lower in the level of achivement).
I am not saying TCS have zero achievements in algorithmic game theory. But TCS loses to economists when game theory is considered.
That's how you can compare the processes TCS. You can't compare the processes of TCS by looking at complexity theory, because TCS is the only community which does it. But you can compare the processes of TCS by game-theory because both computer science and economists do it. As pointed out above, you could also use Quantum Information Systems to measure the effectiveness of the processes of TCS.
The irony is that today's economics systems need CS expertise more than economics expertise, but CS folks are focused on what can be publishable in the next conference. This is not science. The science is what reveals a truly fundamental law in our field of study, whether it is physics, economics, or computer science. Unique Game Conjecture, for an example, is an attempt towards a fundamental law. Non-computability of Nash equilibirum is yet another. How many other TCS game theorists are involved in this science? How's about the four editors of the community book? Economists for an example discovered a fundamental law of individual rationality to base their science.
What is a fundamental law Algorithmic Game Theorists have worked out or are working out? This is risky business, and TCS community does not give us the slack, or the scientific freedom to pursue these risky strategies. Our broader society who pay us though give us this scientific freedom, in terms of granting us tenures. They give us tenure to pursue risky things. What is the corresponding scientific freedom TCS STOC/FOCS culture give? None.
If you do not have a paper in the next stoc/focs you do not matter. That's what the culture is. It is not collaborotive but totally competitive, which promotes hiding ideas until they give to fruition.
Can you give a reference in the economics literature to tolls and congestion?
ReplyDeleteThx.
In ICALP 2004, there is a paper by L. Fleischer about tolls which someone in AGT also once said was already known. But there is no reference to the previous result in the journal version. Does anyone know a reference for the "original" result?
ReplyDeleteEven if something is already known, it can also get into a journal, like some of the work mentioned here. So how does having journals count more than conferences fix the problem?
ReplyDeleteCan you give a reference in the economics literature to tolls and congestion?
ReplyDeleteThis is so straight forward that a self respecting scientists won't even think of writing a paper. Imagine a situation, where the resources are constraint by a linear inequation. How to get the prices of the resources? Anybody having done any linear programming or economics, would tell you, "just take the dual".
When I read TCS paper on toll, where the linear program is written but the dual was not taken, I felt "oh man, this is the career I am jumping in". Ultimately Roughgarden is smart and understands that it is not the depth of the results which would give the tenure but it is the number of STOC/FOCS paper and the name on the editorship of a book.
So as I said, you get what you measure.
If we want to see what TCS did in confront with other areas, if I may go beyond the immediate past -- how about Operations Research?
ReplyDeleteI seem to recall some TCS people with names like Levin, Cook, and Karp helped clarify some very messy ideas folks had about Integer Programming, and Hamiltonian Circuit, and the like with something called NP-completeness.
Also, there was a fellow, Kamarkar, who did stuff with Linear Programming, and even published some of it in STOC/FOCS.
It is not the journal vs conference, but it is the competitive culture of accumulating the most beans which is the bottleneck. Conferences and journals fill in different needs and we want both.
ReplyDeleteMany of us know the questionable JACM acceptance. I especially paid attention to algorithmic game theory, where some of the mediocre algorithmic game theory papers were accepted. Sometimes conferences did a better job, by returning the papers back to the authors.
Janos, I think in combinatorial optimization we were winning. We were bringing better computational understanding of combinatorics and optimization.
ReplyDeleteAs you point out we are all proud of some of the work in combinatorial optimization.
I think we could grant ourselves the score of 2 out of 3. So our total score is 3 out of 9 now. But, keep in mind that this 2 of the 3 we win in combinatorial optimization was earned too far in history. We have a very different culture now. Unfortunetaly, we seem to have deteriorated in the last decade or two.
"Many of us know the questionable JACM acceptance."
ReplyDeleteBut many of us do not. What acceptance was considered questionable and why?
Look at the algorithmic game theory list of JACM for the last 2 or 3 years.
ReplyDeleteThese are JACM game theory papers 2006--present.
ReplyDelete2009:
Moshe Babaioff, Ron Lavi, Elan Pavlov:
Single-value combinatorial auctions and algorithmic implementation in undominated strategies.
Xi Chen, Xiaotie Deng, Shang-Hua Teng:
Settling the complexity of computing two-player Nash equilibria.
Tim Roughgarden, Mukund Sundararajan:
Quantifying inefficiency in cost-sharing mechanisms.
2008:
Heiner Ackermann, Heiko Röglin, Berthold Vöcking:
On the impact of combinatorial structure on congestion games.
Nikhil R. Devanur, Christos H. Papadimitriou, Amin Saberi, Vijay V. Vazirani:
Market equilibrium via a primal--dual algorithm for a convex program.
Xi Chen, Xiaotie Deng:
Matching algorithmic bounds for finding a Brouwer fixed point.
Christos H. Papadimitriou, Tim Roughgarden:
Computing correlated equilibria in multi-player games.
2007:
Anupam Gupta, Amit Kumar, Martin Pál, Tim Roughgarden:
Approximation via cost sharing: Simpler and better approximation algorithms for network design.
Vincent Conitzer, Tuomas Sandholm, Jérôme Lang:
When are elections with few candidates hard to manipulate
Aranyak Mehta, Amin Saberi, Umesh V. Vazirani, Vijay V. Vazirani:
AdWords and generalized online matching.
Andrew Gilpin, Tuomas Sandholm:
Lossless abstraction of imperfect information games.
2006:
Avrim Blum, Tuomas Sandholm, Martin Zinkevich:
Online algorithms for market clearing.
If we had a eprint site with open reviews, many of these issues would have been known to the authors before their papers were published.
ReplyDeleteAlso, again this is not a journal vs conference issue. There is no public penalty for publishing something that is "well-known" by a different community. In fact, it seems to help you. So why shouldn't people keep doing it? How should we incentivize people against it?
Anonymous 60, is how they do it in economics. A good economics paper goes through a life of a working paper and when it survives the adverserial review, which are open and public, only then it is accepted.
ReplyDeleteNeedless to say, a single economics paper could get you a job, tenure, or even a Noble prize.
ReplyDeleteThis is not usually the case with TCS. I do not know any student who did only 1 CS paper and got a job from a top theory school.
Tim Roughgarden didn't get a job largely based on his selfish routing paper with Tardos?
ReplyDeleteRoughgarden has enough beans too and did not get his current jobs until some beans were collected. In any case, Roughgarden started right, but TCS community has managed to make him a bean counter too.
ReplyDeleteTCS has cut his chances of getting a ground breaking result.
"TCS has cut his chances of getting a ground breaking result."
ReplyDeleteWhy? Don't you think that's ultimately up to him? It depends on his own priorities? I don't think you can blame TCS for whether or not he proves fundamental results.
Of course it is upto every person how he/she lives his life. But the community has an effect on everybody by setting the expectation.
ReplyDeleteWould TCS community be fine if he does not produce a STOC/FOCS paper in 2 years in a row? How's about 5 years in a row? How's about 10 years? I think TCS expects at least 3 papers every year. That already effect the problem selection. The criteria becomes, is what I am doing sufficiently easy and interesting to be in the next stoc/focs paper. The community does not give the slack. So if half of your energy just goes to serve the expectations, you can't optimally choose to pick a problem with your own 100% passion.
An ideal community would let a true scientist to choose his/her own problems by let the person himself/herself what he/she is passionate about and what he/she think extends the science, and what he/she thinks is his/her abilities. But TCS culture expects, and our decisions are based on this expectations and not by our own passion. We become a herd and not individual scietists working collaborating.
"I think TCS expects at least 3 papers every year."
ReplyDeleteI'm not sure where this statistic is from. I think it depends on the school. It is well known that at MIT, tenure is usually given for a "big idea" rather than a huge number of publications. Look at some of the people who have recently gotten tenure at MIT, such as Dan Spielman. He did not have a huge number of papers when he got tenure, but he had this "smoothed analysis" idea that had a lot of impact and tackled a hard problem
One example does not say much. If you say Dan Spielman, I will say Moses Charikar.
ReplyDeleteDan in any was not in CS department but in Math department, so some broader ethics of Math also played in.
Look at their CS. Recent theory tenures, have a tons of beans.
The 3 is a random number, I picked. The expectation is regular show up. If somebody collects the actual statistics, I would bet that the actual number would be much more than 3. Take the top 100 TCS folks, and count the number of publications per year.
I am a mathematician and not a TCS regular but have enough interest in the subject to follow this blog. I am totally amazed (as well as somewhat taken aback) about these relevations regarding AGT that I have not been aware of before. Perhaps it is time for some non-anonymous senior expert to come forward and clear the air about this -- that is answer the question whether TCS people are rediscovering paths already well trodden by economists and game-theorists.
ReplyDeleteI feel that in such inter-disciplinary areas the editor or the program committee should be obliged to take expert opinion from the other community. This seems to be common sense, but is this procedure actually followed in practice?
Coming to think of it, I think this problem exists in other more mathematical areas of TCS. For instance, I do not know the exact relations between the work that some TCS people are doing using additive combinatorics and the work of the experts such as Tao, Gowers, Green etc. Given that the latter group are the unquestionable experts, are TCS papers on these topics routinely sent to them for comments (mainly regarding whether the results are already known or trivially deducible from what is known) ?
"Perhaps it is time for some non-anonymous senior expert to come forward and clear the air about this -- that is answer the question whether TCS people are rediscovering paths already well trodden by economists and game-theorists."
ReplyDeleteThe senior "experts" are the ones writing the papers in question ...
What I don't understand is that it is possible that for example, Kleinberg and Tardos did not know that their work was known in the economics community. (I would really like it if the above anonymous could provide a reference for this, which should not be so hard if it as well known as claimed.) If that were the case and other people knew it, then why didn't they say something to prevent follow-up work that also repeats known work from happening?
Coming to think of it, I think this problem exists in other more mathematical areas of TCS. For instance, I do not know the exact relations between the work that some TCS people are doing using additive combinatorics and the work of the experts such as Tao, Gowers, Green etc.
ReplyDeleteI think there is a healthy exchange of ideas and techniques in this particular case. See e.g. http://eccc.hpi-web.de/eccc-reports/2008/TR08-045/index.html and http://arxiv.org/abs/0806.0381v1. Also, see http://www.cs.cmu.edu/~odonnell/papers/dhj-draft.pdf.
Also, the expectation-to-publish problem one of the commentators is referring to is a problem with the grant system, not with a conference system. It has always been publish-or-perish. It may not be a great system, but it is better than giving out grants based on more subjective measures which inevitably leads to more politics.
Finally, I think too much is being made of a few papers reproving known results from a different field. It is at least a sign that we are looking at the "right" questions :)
I think there is a healthy exchange of ideas and techniques in this particular case. See e.g. http://eccc.hpi-web.de/eccc-reports/2008/TR08-045/index.html and http://arxiv.org/abs/0806.0381v1. Also, see http://www.cs.cmu.edu/~odonnell/papers/dhj-draft.pdf.
ReplyDeleteI think the flow is pretty much in one direction. I am yet to see any techniques originating in TCS meaningfully impact research in additive combinatorics (other than nuggets in the intro citing so called "applications" in another "applied" field (i.e. TCS)). But the given the difference in depths of the fields it is probably unrealistic to expect otherwise.
Also, the expectation-to-publish problem one of the commentators is referring to is a problem with the grant system, not with a conference system.
Its not just grants -- jobs, tenure decisions all come into play. I believe one fundamental problem is that TCS is not really a good fit in an engineering oriented department. Many people would like to have a research culture closer to the model in the theoretical sciences or economics. Unfortunately, we have borrowed/inherited this competitive conference culture from the engineers.
I don't see any easy solution other than the painful one of detaching ourselves from Engineering schools and their culture, and try to merge with Math depts if they let us in.
Klienberg-Tardos paper example is a good example, because both of them have the highest standards and did not try to spin things. So one could easily make their paper as examples. There are other examples, which did not form as clean example because the authors spun the results. IMHO, these other authors are way higher culprit. If we mention those examples here, there would be dirty debates, but we know in our hearts that how empty the credit we AGT researchers sometimes get. In some cases we are not serving the science. We are pursuing our personal goals. Ironically, that precisely a problem in AGT. Create a credit sharing mechanism, so that our personal goals align with serving science.
ReplyDeleteThose posters bad-mouthing TCS about originality of contributions may know a tiny piece of the field but they simply don't have any idea about what they are talking about with respect to TCS as a whole.
ReplyDeleteHow about PRIMES is in P? Or the solution of the finite field Kakeya conjecture? (See
Terry Tao's post on the latter.) Or the creation of capacity achieving error-correcting codes in the Hamming model? Or check out the lectures from the last International Math Congress to see Avi Wigderson's or Terry Tao's lectures.
Get a clue!
"I am totally amazed (as well as somewhat taken aback) about these relevations regarding AGT that I have not been aware of before. Perhaps it is time for some non-anonymous senior expert to come forward and clear the air about this -- that is answer the question whether TCS people are rediscovering paths already well trodden by economists and game-theorists."
ReplyDeleteIt is quite tempting to ignore the strange off-topic attack of "anonymous" on AGT. However since I do think that meta-scientific debates are important, and since I do like the blog format, warts and all, as a platform for these, I will reply.
First, we don't really have any revelations here -- just the opinion of one anonymous person who happens to think differently than most other experts. It would be enlightening if somewhere among the lengthy put-downs there was some information. E.g. a link to an economics paper that Kleinberg-Tardos "re-discovered", specific information on what makes the "questionable JACM accept" questionable, etc.
Second, while I am sure that, as in every interdisciplinary field, some results were rediscovered, it is quite ridiculous to claim this as a general characteristic of AGT. Certainly the game theorists think otherwise, e.g., awarding the Shapely lectures to Roughgarden or inviting several AGT people to be on the editorial board of GEB. Many think like (the EIC of GEB) Ehud Kalai that "The most exciting current research in game theory and its applications is being done in computer science".
Finally, I do agree that researchers in AGT as well as in the rest of TCS are publishing too many papers and have written about it in http://agtb.wordpress.com/2009/05/14/readerless-publications/
The last part of the link was cut off; It should be "readerless-publications".
ReplyDeleteThere are more than one anonymous here.
ReplyDeleteAlso, GEB is not a top rated economics journal. I think you know this.
ReplyDeleteMany think like (the EIC of GEB) Ehud Kalai that "The most exciting current research in game theory and its applications is being done in computer science".
ReplyDeleteAny link?
On JACM papers:
ReplyDelete>Vincent Conitzer, Tuomas Sandholm, Jérôme Lang:
When are elections with few candidates hard to manipulate
Anyone who reads this one will agree that this is unspeakably embarrassing.
Anyone who reads this one will
ReplyDeleteagree that this is unspeakably
embarrassing.
The JACM referees didn't, did they?
It's a pity that despite numerous calls for giving more specific references, the main person(s) critisizing AGT papers is still unable to substantiate the claim that the results from the paper by Kleinberg and Tardos have been known to other community for a long time.
ReplyDeleteIt is a pity that people want another word by word reference of Klienberg Tardos paper before accepting that it is a rediscovery. If that were the case, it would be called copyright infringement.
ReplyDeleteOpen any standard book on the subject and be content with. If you are looking for a paper which says word by word what KT paper said, then your standards of scientific novelty are too low, though still may be higher than AGT folks.
Re #77 "There are more than one anonymous here.": this is why, while I think anonymous commenting is a good thing, I also think it would be helpful if some of the anonymice made up nonce names for themselves so that we could distinguish them from each other.
ReplyDelete|On JACM papers:
ReplyDelete|
|Vincent Conitzer,
|Tuomas Sandholm,
|Jérôme Lang:
|When are elections with few
|candidates hard to manipulate
|Anyone who reads this one will
|agree that this is unspeakably
|embarrassing.
|The JACM referees didn't,
|did they?
I am amazed by the high browing comments here. Conitzer et al is a great paper, the NP hardness proofs are marvels, and the algorithms reek of original techniques. AI people, social sciences people absolutely love that paper. It will be a classic in 10 years, frankly, I am amazed it did not get into STOC or its weaker cousin FOCS.
I am not sure if Anonymous commenting is a good thing. There are so many genuine complaints anonymous make, instead of fixing them or even acknowledging them, a senior person like Noam comes and say everything is alright. They just like to protect their powers. Did Noam point out any specific work besides what anonymous have acknowledged is great in AGT, such as price of anarchy and complexity results.
ReplyDeleteDid Noam say, this is what AGT folks are doing which will help change the world? He only acknowledged that people are writing a lot of publications. If AGT folks are doing great work then is not a lot a good thing? It feels like in his heart he knows AGT work is big in quantity and not quality. Somebody posted JACM list, half the papers are embarrasing. In fact, editors of JACM have accepted papers which did not get all positive reviews. It was done just to raise AGT beyond what it is worth. The expert reviews were ignored and often obvious experts when gave bitter reviews were not asked for their opinion. It was made absolutely clear, either give good reviews or not at all. This I know based on insider account and not based on hearsay. The only clean school in AGT seem to be Berkeley. In some sence Berkeley has the highest standards of pursuing science. As the earlier post revealed, those scientists work for a fraction of money than their "Institute of Technological" counter parts. Luca is right that funding hurts Berkeley becoming even better. Case in point. Madhu Sudan a giant in TCS considered the west coast, now sitting in Microsoft. Given the funding constraint at Berkeley, I now have more respect for their excellance!
"Open any standard book on the subject and be content with. If you are looking for a paper which says word by word what KT paper said, then your standards of scientific novelty are too low, though still may be higher than AGT folks."
ReplyDeleteWhat is a "standard book" and in what chapter? If it is so well known, why can't you just give a reference? Even if it is not the same wording of the result, give a reference and explain how the result is implied.
Thx.
I am being amazed by the critic regarding papers accepted to JACM...
ReplyDeleteIt also seems a quite large list of papers from AGT published in JACM in a short period.
About Kleinberg-Tardos paper, as I heard recently there is a nice paper by Bateni, Immorlica, Hajiaghayi, and Mahini who show how these results (and I think the follow-up FOCS paper) are almost immediate from economy literature and then they resolve some open problems in both fields of cs and econ.
ReplyDeleteAGT area editors of JACM will tell you how this long list, some are great and some mediocre, got accepted in JACM. I think the reviews of the accepted paper should be made public. This way the community could see when the stadards are lowered, and ask for the reasons.
ReplyDeleteIt is clear that some of the anonymous are senior folks too. The TCS community would be transparent if people, or at least the senior folks, could feel free to put their names to criticizing comment. Or when senior people do put their name, they do not act as puppet but humans acknowledging criticism and also write the positive things. Noam comment became meaningless because he tried to be one sided and protective. He does not look balanced.
No links, no tails ... isn't this a result of what I've said before: HARD competition, dependence of someones subjective opinion, saying what you really think to a "giant" could be a "sentence of depth" for your carrier ... Does this (small excerpt from this local discussion) not explains why CS conferences are such as they are? And why "having free mind" can imply "having no job?"
ReplyDeleteActually I don't understand quite well all this excitement on some (if any) re-discoveries? There are a lot of results rediscovered several times. What is here so bad? Bad is, of course, that we not always know all what was done. But who knows that for sure? What if this (not being sure that something similar could be already shown by someone) would be a reason for not to show your own proofs -- where would be then? Should we keep our proofs in our drawers until it is "crystal-clear" that nothing similar was published before? Who should recheck this? The FBI?
http://www.ece.northwestern.edu/~nickle/research.htm
ReplyDeleteThe paper does not appear to be available ... so still no reference.
ReplyDeleteStacy, the bad is not rediscoveries. It is just used as an example. The bad is not letting people realize their full potential. The bad is using the names on the paper to judge the papers. The bad is knowingly hiding simple proofs for the complex proofs. The bad is only working for a paper and not for the scientific enlightenment. The bad is not making an attempt to do a literature search.
ReplyDeleteThe reason behind all this bad is not only our conference culture, but an arbitrary review process which is opaque and accountability less.
"Stacy, the bad is not rediscoveries. It is just used as an example."
ReplyDeleteYes, this clear. "Big" damages are best seen on examples. And I agree with what you said later.
But I cannot completely agree with
"The bad is not making an attempt to do a literature search." Not trying to search is bad, no doubts. But we do this! We try to do this! However, we live with "google" or "google scholar", etc. And they just show us a "pot", not what the "soup" is there. So, we should trust much more to our "older" colleagues. They know much more. But how can this be realized within a HARD competition? This is the problem.
frankly, I am amazed it did not get into STOC or its weaker cousin FOCS.
ReplyDeleteAs an outsider I have always thought that STOC and FOCS were THE top two CS conferences of equal standing.
Is it true that there is indeed a diferentiation between even these two in terms of prestige ?
"As an outsider I have always thought that STOC and FOCS were THE top two CS conferences of equal standing."
ReplyDeleteThey are, of course. Or better to say: they *were* such. New trends enrolled in the meanwhile ...
"The bad is only working for a paper and not for the scientific enlightenment."
I agree completely. Not only I: see Noam's very nice remarks on this.
I agree with Lance that there is a big problem of the significance of conferences pushing people in CS towards rushed publications and conference proceedings that represent the "smallest publishable unit".
ReplyDeleteHowever, I don't think that paper journals are the solution. The editorial boards of journals tend to be conservative and reject work that is new and different.
I believe that the fundamental problem is that good and significant work takes time to mature. Rob Schapire and I are finally finishing a book about boosting which was 10 years in the making, and I am still not quite satisfied with the way the more recent work on boosting is described in the book.
This long maturation time can result in problems of assigning credit where credit is due. There has to be some way for people to publish papers that are at a relatively initial stage and secure their credit. The solution that I use is to publish papers on Arxiv. This is a way of making papers available to peers while maintaining a record of the contributions of yourself and your students.
Then comes the issue of publicity and quality assurance. It is not enough to make papers available, there are just too many papers. There have to be mechanisms for helping people find good papers and for helping authors find receptive audiences. This need is supposed to be filled by conference program committees and by the editorial boards of journals. However it is clear that these mechanisms are not sufficient and new mechanisms are needed.
I have started a web site whose goal is to provide such a mechanism for machine learning and related areas. The name of the web site is
themachinelearningforum.org.
The basic idea is that anybody can submit a link to a paper and that people with a proven track record can write reviews of papers (but not of their own papers). The reviewer of a paper is not anonymous, so people get some credit for writing a good review.
This site has been up for about a month now, and we are still working on it. Initial response has been very good, but it remains to be seen if it will take on the role I hope it will have. If you are interested in Machine Learning, please register to the site and contribute.
It would be a relatively easy thing to start a similar site for theoretical computer science. I would be happy to help.
^^^^
ReplyDeleteI like Yoav Freund's idea of a forum for TCS very much. Blogs are directed by a single (or, in this case, a couple of) individuals. It would be an interesting experiment to have multiple threads controlled by a community. This would be especially valuable for people who are not in major research institutions, and it would give a source of feedback, other than conference PC reviews, about the significance of results.
I can't add much after 100 comments, but:
ReplyDeletereply to Paul Beame: what is wrong with (T)CS adopting the practice of mathematics, which attacks 1 - 5 as follows?
1) arXiv (why are there so few CS papers there?)
2) invited addresses, workshops, special sessions, prizes, Mathematical Reviews, blogs (in particular, why is Computing Reviews so little used in CS?)
3) journals
4) open source, followed by society journals (there are problems with free-to-read-but-not-to-write journals as is becoming common, but progress is possible)
5) electronic journals (arXiv overlays?)
Reply to aravind: a concrete suggestion for improving papers - editors should require that each paper explicitly contain a literature review and a section called "our contribution" that explains how the present paper improves on the past, and why we should care about such an advance.
Requiring people to referee in proportion to their publishing would certainly reduce the number of papers, especially if their referee reports could be rejected if challenged as inadequate by the authors.
To answer Mark Wilson's question of "why not the math model?":
ReplyDelete(1) Speed: arXiv (why are there so few CS papers there?)
This is largely historical. For quantum, computational geometry, and crypto the arxiv is very active. For complexity there has been an alternative ePrint server (ECCC) and in the algorithms and complexity areas the arxiv was initially dominated by work that was peripheral to the field and crap submitted by people who didn't know what they were talking about. This inhibited people with good papers from submitting there. This may change and certainly the arxiv is a good model for speed (though certainly not for its user interface for readers) - I said as much in my earlier comment.
(2) Selection:
invited addresses, workshops, special sessions, prizes, Mathematical Reviews, blogs (in particular, why is Computing Reviews so little used in CS?)
The very jumbled nature of this suggestion shows why it doesn't cut it. We already have workshops in addition to conferences but they only help a small minority who can attend them. We have a few invited talks but this is hardly enough to cover the speed and directions in which the field is moving. The typical math conference has one to two plenary invited talks per day (still too few to highlight the new directions since they are often designed to showcase stars rather than simply the best recent work - though the two often go together), a number of sessions in which organizers invite their friends or people whose work they know, and a collection of other contributed papers with wide variation in quality.
My experience has been that the most interesting parts are in the invited talks and organized sessions but they don't do much of a job at highlighting the range of the best new work. (I hesitate to emulate math conferences on another level: while there are many good expositors in math, why are there so many math talks in which the authors print out sections of their papers on slides and proceeded to read from them?)
As for Computing Reviews - when you have something of low quality it tends to keep the good work away. To all intents this died out in the 1980's. The level of diversity of all of CS is an order of magnitude greater than in mathematics, which makes the Reviews model unworkable. It is a fairly slow process, too.
BTW: There is a separate debate on society journals versus open access journals. Open access journals (along with the arxiv) win on availability but longevity may be an issue for those that do not have a "pay-to-publish" model. From the point of view of serving their members the ideal would be a society-published open-access journal. I guess there is the problem that someone has to pay the headquarters staff.
I would be very interested in having the anonymous commenter re the K-T paper substantiate with references to book chapters or articles currently available online or at a university library.
ReplyDeleteHaving spent a good amount of time analyzing the computational complexity of related problems, I find it hard to believe that the econ theoretic techniques, however simple they are after the fact, are obvious to apply to the K-T collaboration problem unless indicated with the problem set obviousness of "use technique A to solve problem Y".
Simpler techniques to solve a problem can be better for many reasons, but that doesn't mean they are apriori obvious.
"I think the flow is pretty much in one direction. I am yet to see any techniques originating in TCS meaningfully impact research in additive combinatorics (other than nuggets in the intro citing so called "applications" in another "applied" field (i.e. TCS))."
ReplyDeleteThe "quadratic" Gowers inverse theorem was proved independently by both Samorodnitsky and Green+Tao. Samorodnitsky's theorem works for functions over even characteristic, whereas Green and Tao's version works over odd characteristic. Both proofs follow Gowers' approach and so bear the same structural outline. Both versions have applications in their respective field, the former in constructing PCPs (a followup work with Trevisan), and the latter in obtaining stronger bounds for Szemeredi's Theorem of length 4. However, neither calculation seems to work if one switches the parity of the field's characteristic.
Subsequently, the "cubic" Gowers inverse conjecture (over finite fields of low char) was again independently shown to be false by Lovett+Meshulam+Samorodnitsky, and Green+Tao. In fact, key steps in that paper by G+T use a lemma by Bogdanov+Viola (Tao has a blog post on this) and a neat Ramsey-styled argument from Alon+Beigel's CCC paper. L+M+S gave a stronger bound in measuring the "dis-correlation" of S_4 with cubics, though G+T's argument may be easier to visualize.
As a previous commenter noted, Dvir's short proof of the Kakeya conjecture over finite fields uses the "polynomial method" in extremal combinatorics, which is frequently used in complexity and coding.
Consider also (dense) graph property testing. While Szemeredi's regularity lemma already implies several corollaries about graph testing, many papers since 2000 have made more progress and develop new techniques.
As an application, consider the "arithmetic regularity lemma" paper by Green. He conjectured a particular form of a removal-type lemma holds for function. As a special case, he proved via Fourier analysis a "triangle-removal lemma" for functions. Last year, Kral, Serra, and Vena, using a particular form of Szemeredi's regularity lemma developed in property testing, gave an non-Fourier-based proof of Green's triangle removal lemma. Since the proof is non-Fourier, their work generalizes to settings with more flavor. Building on their techniques, both K+S+V (in a followup work) and Shapira, in independent papers, resolved Green's conjecture affirmatively. Though I haven't parsed through their proofs, they do seem to rely on perspectives from both additive combinatorics and property testing.
Victor Chen
Much of the discussion has focused on TCS (algorithms, complexity, semantics etc) and ignored that Lance's original claim was about ALL of CS. Similarly comments on the Snyder et al CRA report ignore that it was specifically limited to "experimental CS" (eg OS, DB, networking, architecture...). Throughout any discussion on conferences, journals, research ranking, job prospects etc, we all need to realize that different fields within CS work very differently. I urge theory people to look at descriptions of the top systems conferences eg http://matt-welsh.blogspot.com/2009/07/sensys-2009-pc-meeting.html to see an alternative model from FOCS/STOC that is also not like Math, Physics or other old fields!
ReplyDeletePaul Beame wrote:
ReplyDelete"I hesitate to emulate math conferences on another level: while there are many good expositors in math, why are there so many math talks in which the authors print out sections of their papers on slides and proceeded to read from them?"
While this may be so (but this is changing with growing use of the latex package beamer), 90% of TCS talks are devalued and rendered contentless by meaningless pictures/animation, as well as attempts at stupid humor/lame jokes etc. One should understand that giving a scientific talk is a serious business (almost as serious as writing a paper), and the limited time should be used very judiciously to transmit as much information as possible. Mathematicians often take this advice too literally, but CS talks often go the other way and become frivolous and devoid of scientific contents for the most parts.
To #105: conference talks may be serious business but that doesn't imply that they ought to be boring. I find your rejection of pictures alarming — illustrations can be very helpful not only for conveying information more concisely than words but for keeping the audience awake. Even clip art can help in differentiating the slides from each other and keeping them memorable, so that the audience can better remember later in the talk what happened earlier. My preference would be to have a picture of some sort on every slide of a talk, although my talks usually don't quite reach that standard.
ReplyDeleteRe: One should understand that giving a scientific talk is a serious business (almost as serious as writing a paper), and the limited time should be used very judiciously to transmit as much information as possible.: no. This attitude leads to horrible unlistenable talks in which the authors goes on and on about the minutiae of his or her paper and the slides are full of unreadable tiny print. One should carefully edit down the content to a suitable subset of highlights that will convey the nature of the talk, and lead interested audience members to go read the proceedings if they want to find out the details. Too little is better than too much.
Carter, simple and trivial are two different things. The algorithm in KT was perhaps not known in the economics literature. Given the existing work in CS and ORabout flows, the algorithm could be given as an excercise in an advanced grad class.
ReplyDeleteThe economics part is what anonymous are disputing here. One thing to note is that this paper is still better than many others in AGT. In those papers you could feel an intentional spin.
Illustrations can be very helpful not only for conveying information more concisely than words but for keeping the audience awake.
ReplyDeleteThis idea that its the speaker's responsibility to keep the audience awake and entertained (and might be to slip in some useful scientific tidbits in between that the audience members can sub-consciously absorb) encapsulates much that is wrong in our community.
It is the primary responsibility of the speaker to give a clear talk with legible and organized blackboard work (or transperencies if that is the case). But it is also the responsiility of the people in the audience to pay utmost attention and to think; and not expect entertainment or sound-bites. A scientific talk is not a sit-com.
This undue expectation of being entertained on every occasion has seeped into our classrooms and in our textbooks, diluting content and leading to a point where students (I am talking about US undergraduates mostly here) expect to be entertained more than being lectured to. The results are there for all to see.
I wanted to comment on Lance's point that conferences aren't any longer used to bring the community together, and that switching to a journal system would restore conferences to this role.
ReplyDeleteAs a graduate student, I have had many opportunities to travel to meet people at conferences where I went to present my work. If we switched to a system where we only published in journals, and conferences were used for socializing, I fear that graduate student attendance at these conferences would suffer. As a result, when the community is brought together, many graduate students would be left out.
In the current system, a graduate student who gets a paper into a conference can usually find a way to go - perhaps from his advisor's grant, a travel award, an internship, his department, etc. (And it has not always been easy for me, and I even had to pay for some conference travel out of pocket.) I think a graduate student would have a harder time finding funding to go to a conference to "socialize."
Moving to conferences-for-socializing system would probably have disparate effects on students, researchers, and faculty, and this should be taken into account.
Moving to conferences-for-socializing system would probably have disparate effects on students, researchers, and faculty, and this should be taken into account.
ReplyDeleteThis does not have to be so. How do mathematics graduate students manage for instance ? I think if we have a annual meeting (in the style of say the winter AMS meeting) departments will be happy to provide travel support (especially since there will be only nominal registration fees). Moreover, we could have organized job fairs/interviews for faculty positions at smaller colleges at these meetings in the style of AMS meetings. So it would provide even for incentives for graduate students to attend.
Re "Departments would be happy to pay for travel support": your department, maybe. Mine has no money for that sort of thing, and making a grand reorganization of CS conferences isn't likely to suddenly provide them with money.
ReplyDeleteEspecially if the trip was overseas and therefore cost that much more per attendee. You weren't assuming that CS happens only within the US, were you?
I thought conferences are also for the presentation of papers, in addition to socialization. Anyone who has discovered something new recently should be allowed to present without a selection process that rivals college admissions.
ReplyDeleteSee also Dan Reed's "Publishing Quarks: Considering Our Culture" at
ReplyDeletehttp://www.cra.org/CRN/articles/march09/Reed-Musings.html
One issue to consider that hasn't been raised yet -- many top conferences in CS disallow simultaneous submission of the same paper to other conferences (reasonable) and to journals (not so reasonable).
ReplyDeleteFor me, this often creates a question of conference XOR journal.
What's the benefit of not allowing a paper to be under review at both a conference and a journal at the same time? If we're to make a transition from being a conference-centric to a journal-centric discipline, ceasing this restriction would be a good initial step.
The Computer Graphics community has moved to a model where the best conferences have their proceedings published as a special issue of journal. For ACM SIGGRAPH (conference) and ACM Transactions on Graphics (journal) this has resulted in an increase in prestige for both. At least six other significant graphics conferences have their proceedings published as special issues of journals (Eurographics, CGI, SGP are three). Alongside this, some conferences are introducing more non-published talks, bringing back the ability to have work-in-progress presented and discussed without the need for an archival quality publication attached to the talk. ACM SIGGRAPH, for example, has so much stuff alongside the technical papers that it is quite possible to fill the entire week going to useful sessions which have no archived publication. In addition, ACM SIGGRAPH now invites the authors of the best ACM Transaction on Graphics journal papers to come and give a presentation about their journal paper. All of these ideas could be picked up by the rest of the CS community and all of them allow a gentle progression towards a better situation rather than wishing for some magic bullet solution.
ReplyDeleteI'm cross-posting this comment I left at the CACM article:
ReplyDeleteYou've described a problem that is dauntingly large and hard to solve. We can make progress by tackling pieces.
The database community has one piece of the answer: their premier conference, VLDB, is now driven by a journal submission model (http://www.jdmr.org/). Authors submit to the journal, and a year's worth of journal submissions are presented at each year's conference. The above url links to an excellent discussion of and rationale for their procedure.
You also highlight the way the proliferation of conferences has led to a breakdown in their value as tools for networking and drawing the community together. For these, I propose a simple solution: colocation. Take all those little conferences, and hold them all at the same time under one roof. Let them keep their independence, but share their coffeebreaks and banquets. Most importantly, allow anyone paid up at one conference to attend all of them---after all, no matter how many conferences someone "attends" at the same time, they can only consume one human-being's worth of resources. The federated computing conference does this in large, but it would be equally affective for subfields of computer science to bring together all their sub-subfields.
Lance, I understand your essay as arguing that conferences cause the publish-or-perish syndrome in computer science. But all academic fields seem to suffer from it. Why blame the conference system?
ReplyDeleteI'm all in favor of growing up. However, before that journals must be fixed. As others (e.g. Aspnes) said I don't understand why we write and review the papers, and then pay a publisher to read them. Also journal reviewing is broken: If I am on the PC of some conference, that at least gives me a line in my CV. Why should I do journal reviewing? Not so sure.
ReplyDeleteSo, please, fix journals before promoting them...
// Roger
Since I largely fund my own travel out of a meager graduate student stipend, this is has led directly to money-based decisions when I send a paper to one conference over another.
ReplyDelete