Friday, August 28, 2009

CS and the Web of Knowledge

Often the best papers from a particular conference are collected into special issues of a journal where they go through a traditional journal review process. Having a paper in a special issue is a bit more prestigious than a regular journal paper.

Recently I co-authored a paper invited to a special issue and we had to turn them down. Why? For that we have to talk about the Web of Knowledge.

I have generally ignored the ISI Web of Knowledge, an subscription-only index of academic literature by Thomson Reuters. The web of knowledge didn't index CS proceedings or tech reports, so sites like Citeseer, DBLP and Google Scholar were much more useful.

Unfortunately, I can't ignore ISI as easily as I'd like to. Both Northwestern and the National Science Foundation use the database to pre-populate the publications in my on-line annual reports. I'd have to manually add my conference papers. Finally over the summer ISI has added most CS conference proceedings papers so this process ought to be easier in the future.

But that's all minor compared to what I have seen happening in some European countries where the ISI is taken way too seriously. ISI has different paper types: Articles, Proceedings and a few others. Some countries, which use these numbers for hiring, promotion and grants, are just counting Articles which puts computer science at a comparative disadvantage where say STOC papers don't count.

Now to answer the question about special issues: The ISI labels special issue papers as "Proceedings" so they don't get labeled as true articles and wouldn't help my co-author, who needs more "Article" papers. So we turned down the special issue for rather technical reasons.

31 comments:

  1. This seems like a rather silly reason (can't you just change it after it gets pre-populated?) and the article feels more like self-boasting on things rather than an informative discussion.

    ReplyDelete
  2. Depressing. I would think that given your relatively high status, you would not have to worry about "paper counting" and "upgrading conference papers to journal papers". And labels such as "proceedings" or "article".

    I understand that you probably mean to watch for the interests of your (lesser?) coauthors, but still... your blog post is depressing.

    Frankly, we ought to make sure we *read* papers before judging people. You should be denouncing people who do otherwise. Anyone who pretends to evaluate another researcher without reading/browsing the papers, is a fraud.

    Sure, Stephen King wrote a lot of very large novels, but that is not why he is good. Good people tend to write a lot of papers, but they are not good *because* they write a lot of papers. I am tired to see causalities and correlations being systematically mixed up.

    A researcher is good because he provides special insights in his research papers. This happens to be correlated with the number of his papers, and sometimes with the relative prestige of the venues, but please...

    Let us reward quality work, not quantity or labels.

    ReplyDelete
  3. Yep. Depressing.

    If the job market were better, my response would be purely economic: Any department stupid enough to rely so heavily on a faulty metric to judge its faculty deserves the faculty it keeps!

    ReplyDelete
  4. Speaking of faulty metrics by which to judge the quality of faculty research by, here is one even worse (i.e., having IMHO a lower correlation with the quality of research) than paper count: number of grant dollars awarded.

    I do know of departments that use this metric, and Jeffe's parting remark certainly applies to them.

    ReplyDelete
  5. The report by Meyer et al (representing a consortium of European CS departments) in CACM on evaluation criteria for CS research makes strong ammunition against people who think ISI is useful for CS. It explicitly says to avoid it.

    ReplyDelete
  6. Nice post. I don't even know that one can manually add papers into ISI.

    ReplyDelete
  7. Why do some professors coerce their students into putting their names on their papers when their contribution is minimal, except in the form of grant money?

    Abolish counting altogether, and we might see an improvement to such unfair and evil practices.

    As a general rule of thumb, a person is a co-author if and only if he/she has proved a theorem in the (theoretical) paper. All other contributions should be mentioned in the acknowledgements section. That's what it is for.

    ReplyDelete
  8. Why do some professors coerce their students into putting their names on their papers when their contribution is minimal, except in the form of grant money?

    This has a name: ghost writing. It is a form of *plagiarism*. It is common. Widespread. Unethical. And yes, it is a side-effect of paper counting.

    And no, I don't think paper counting is unavoidable or a necessary evil. We only accept it because we have become too cynical to care.

    ReplyDelete
  9. Beyond the general regret about these non-scientific considerations, I also doubt that this strategy makes sense just from a selfish-rational point of view: The exact way you describe that NSF/ISI/(some university) treats special issues is an artifact and will most likely change (likely several times by several of the involved parties) in the not so far future. Stretigically responding to this artifact is thus likely not rational in the long run.

    ReplyDelete
  10. I believe this "special issue" phenomenon is somewhat peculiar to the CS (having to do with the conference sub-culture). My experience is that special issue articles undergo a quicker but also a substantially lower level of scrutiny from the referees (compared to ordinary journal submissions) . Also, such special issue papers are handled almost exclusively by the "guest editors" without much oversight from the journal editorial board or even the editors-in-chief -- the guest editors are under no obligation to maintain the standards of the journal. For these reasons I think it is correct to label articles appearing in a special issue as conference proceedings papers. They should certainly not be counted as journal papers. I do not understand why anyone would consider them to be more "prestigious" -- just because they got the imprimatur from a conference program committee ? Also, we are back to the conference vs journal debate ...

    ReplyDelete
  11. Why do some professors coerce their students into putting their names on their papers...

    I know this happens, but I think students who let it happen to them are fools. If a professor unethically requests coauthorship, just tell them no. Their reputation is worth more than the coauthorship, so they should leave you alone.

    ReplyDelete
  12. Anonymous #10: my experience with special issues is the opposite: they undergo the normal level of review but are even slower than normal journal papers because one has to wait for all the other papers in the same issue. I don't think having a guest editor in place of the usual editor makes much difference in the overall editorial standards.

    ReplyDelete
  13. um... What? I have never heard of a special issue being more prestigious than a regular issue. And special issues that are collections of papers from proceedings are certainly not more prestigious. They're simply conference papers and should be counted as such.

    Your real complaint appears to be with the fact that paper counting doesn't take into account that in CS, conference papers are at least, if not more reputable than journal pubs.

    I have, several times, heard the advice that I should avoid the temptation to submit to these invitation journals. If there is substantial material beyond the conference version of a paper, submit it to a regular high-end journal. If not, you already published it!

    ReplyDelete
  14. Anon #10: The editors and editor-in-chief do have some input with respect to special issues. They decide (or at least the editors-in-chief do) if the journal will publish the special for the conference based on the quality of papers that, historically, appear at that conference.

    Remember, these are the top 10 or so of the hundreds of papers that were submitted to the conference. If the conference and journal are appropriately matched then this should make them some of the best papers that appear in the journal.

    ReplyDelete
  15. And special issues that are collections of papers from proceedings are certainly not more prestigious.

    Generally they are not "collections of papers from proceedings". They tend to be the top 10-20% of the papers from the conference by invitation only. Having served in many committees, invited papers do carry special weight though not necessarily over other issues of the journal, but as a further distinction on your conference paper (considered to be in top 10-20% of conference).

    They're simply conference papers and should be counted as such.

    Generally authors are expected to revise and furnish more complete versions of the paper for the special issue.

    I have, several times, heard the advice that I should avoid the temptation to submit to these invitation journals.

    Aside from the ISI factor and the delay in publishing, generally the contrary is the case: do accept invitations to special issues.

    Say for example you might have a top 10% of STOC/FOCS/SODA paper which is not yet good enough for the rarified (non)standards of JACM (as few papers are), yet if submitted to SICOMP as a regular paper would not be seen as particularly special. However a top 10% distinction from those conferences would definitely carry weight with an evaluation committee and rightly so.

    ReplyDelete
  16. My experience is that special issue articles undergo a quicker but also a substantially lower level of scrutiny from the referees...

    Not only that, but I can tell you from my own experience that even if a referee says "you know, this isn't actually such a good paper after all..." the paper must be accepted anyway once the committee has chosen it.

    ReplyDelete
  17. even if a referee says "you know, this isn't actually such a good paper after all..."

    Papers usually have more than one referee, so a single negative comment won't sink it if the other referees are sufficiently positive.

    the paper must be accepted anyway once the committee has chosen it.

    I know of several papers rejected from a special issue. They are, as one would expect, rare circumstances since by now people agree the result is top 10% but it does happen.

    ReplyDelete
  18. Anonymous wrote "I know this happens, but I think students who let it happen to them are fools. If a professor unethically requests coauthorship, just tell them no."

    I think you are a fool. You expect a student to define what is "ethical" when I think faculty working together would differ on the definition, and then you want them to sabotage their relationship with the person who is going to write their next letter of recommendation if they don't feel the professor "did enough" to merit being a coauthor.

    ReplyDelete
  19. their next letter of recommendation if they don't feel the professor "did enough" to merit being a coauthor.

    Students tend to underestimate the contribution from their supervisors.

    See the related thread at FSP:

    http://science-professor.blogspot.com/2009/08/adviser-with-benefits.html

    ReplyDelete
  20. Yes most advisors are not abusive, just as most people are not criminals. We have a system to find out and hold those criminals accountable for their crime.

    We do not yet have a system for finding out abusive advisors and do not care to hold them accountable. Some are even highly reputed professors. If a student speaks up then the student is held accountable for speaking up.

    ReplyDelete
  21. Students tend to underestimate the contribution from their supervisors.

    This must be written by a supervisor. You seem to be saying that students tend to have poorer judgment/memory about who contributed to what than professors do.

    I suggest you read the first letter of this edition of "Ask Professor Nescio" published in the AMS Notices:

    http://www.ams.org/notices/200805/tx080500590p.pdf

    ReplyDelete
  22. You seem to be saying that students tend to have poorer judgment/memory about who contributed to what than professors do.

    Yes, I'm saying exactly that. Students usually have a very accurate tally of who did the handiwork, but they lack the perspective to evaluate high level contributions from their supervisor, such as suggesting a problem, suggesting a line of attack, proving key lemmas when the student gets stuck, etc.

    This is not to say that there aren't cases where supervisors get wholly undeserved authorship or that in some instances such high level contributions do no raise to the level of meriting coauthorship.

    ReplyDelete
  23. Students (...) lack the perspective to evaluate high level contributions from their supervisor

    Then I propose that a note be added to the paper stating "Whereas the paper was written by StudentX, the second author, ProfessorY, made high level contributions through valuable discussions and advising."

    ReplyDelete
  24. "Whereas the paper was written by StudentX, the second author, ProfessorY, made high level contributions through valuable discussions and advising."

    In the abstract I agree. It would be nice to have a movie-credits like list of authors: Writing by X and Y, high level ideas by X and Z, Experiments by Y, etc.

    In practice it wouldn't work. Memory is too imperfect. For example, there is this paper where a key idea is used. They way I remember, the student suggested it. The student swears it was the other way, that I suggested it.

    In another, I suggested and developed an entire line of attack (this was with another professor) but then got stuck in one relatively minor last step. My colleague jumped in and resolved the issue with some clever arithmetic manipulations. Is that high level? low level? I don't even want to know. I was glad to publish the paper jointly. Oddly enough, in my experience some grad students would have resented this, had they been in my place.

    ReplyDelete
  25. It should be pointed out that this is not a graduate student/advisor issue. Apparently some professors also frown upon the practice of co-authorship with minimal or no effort.

    http://www.ams.org/notices/200707/tx070700822p.pdf

    ReplyDelete
  26. This comment has been removed by the author.

    ReplyDelete
  27. In the abstract I agree. It would be nice to have a movie-credits like list of authors: (...) In practice it wouldn't work.

    The fastest growing international research journal does precisely what I suggest. Here is an example taken out of the PLoS One journal:

    Author Contributions

    Conceived and designed the experiments: HN. Performed the experiments: HN. Analyzed the data: HN. Contributed reagents/materials/analysis tools: HN DK MR JAPH.
    Wrote the paper: HN DK MR JAPH.


    Reference: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0006852

    Notice how there is a bit that says "Wrote the paper". Well, if the paper was indeed written by the student, we can say so right there.

    Oh! And this can be done with existing journals and conference proceedings, right now.

    ReplyDelete
  28. Those commenting that Special Issues are somehow less than regular journal issues simply haven't had the experience of CS journals. In a culture that values conference publications highly relative to journal publication, the highest is for invited conference special issues. The key point is that these are invited special issues. This is not just in theory but in all of CS.

    For years, JCSS published special issues for many major CS conferences but its regular papers were of much lower quality (after all such papers tended to fill out the spaces between the special issues). A senior member of the field in the 1990's told me the following pecking order of a selected few CS theory journals:

    J.ACM
    > JCSS special issue
    > SIAM Journal on Computing
    > Information and Computation
    > JCSS regular issue

    ReplyDelete
  29. P.S. I forgot to mention that the critical part is that these special issues be invited from already highly selective conferences. Obviously the strength of the special issue depends on the conference itself. A strong journal will only choose special issues for strong conferences.

    ReplyDelete
  30. Mr. Lemire. On your blog you have in the terms of use "If you think I am an idiot, please post it elsewhere."

    Is this the sort of elsewhere that we can say if we think you are an idiot?

    Just wondering.

    ReplyDelete
  31. I agree with Anonymous 22. In fact, I have seen the best PhD advisors use the socratic method to great effect-- and if they do it right, the result is that the graduate student thinks they accomplished that first paper "all by themselves"! As the graduate student matures, they end up doing more and more themselves. Sometimes, they even figure out how much their advisor helped them in the beginning years later :-)

    Now, this doesn't apply to all situations, and there are definitely situations where advisors inappropriately don't take themselves off their students' papers, but it happens more than you might think.

    And: while it is inappropriate in a theorem-based paper for someone to get co-authorship only for funding, I note that it is the expected norm in most of the experimental sciences. So I agree with the comments on this so far, but remember, it is not portable to other fields.

    ReplyDelete