Thursday, August 27, 2015

PACM

I serve on the conference committee of the ACM publications board and we've had extensive discussions on the question of the role of journals in publication venues. A number of CS conferences, though notably not in TCS, are moving to a hybrid publication model where their conference presentations make their way into refereed journal papers. One of our proposals is the creating of a specific venue for these activities, a new Proceedings of the ACM. In the September CACM,  Joseph Konstan and Jack Davidson lay out this proposal, with pros and cons by Kathryn McKinley and David Rosenblum respectively. The community (that means you) is being asked to give their input.

The theory model has not significantly changed since I was a student. Papers submitted to a conference get reviewed but not refereed, the proofs read over usually just enough to feel confident that the theorem is likely correct. Once authors started submitting electronically they could submit entire proofs, though often in an appendix the program committee is not required to read.

The papers appear in a proceedings and to quote from the STOC proceedings preface
The submissions were not refereed, and many of these papers represent reports of continuing research. It is expected that most of them will appear in a more polished and complete form in scientific journals.
A small select number of papers from a conference are invited to a special issue of a journal where they do go through the full referee process. Some, but not most, of the other papers get submitted to journals directly. We don't have the proper incentives for authors to produce a journal version with full and complete proofs.

Should theory conferences move towards a more hybrid or PACM type of model? I'd had several debates with my fellow theorists many of whom feel the advantages of requiring journal-level papers get outweighed by the extra effort and time required by the authors and the reviewers.

8 comments:

  1. I once heard the following from a famous professor: Journal papers are pointless. If a result is important enough, then someone will put it in a textbook and work out the details. If it's not important enough, then why should anyone waste time working out the details.

    ReplyDelete
    Replies
    1. Supposedly the authors already worked out the details, so I don't see how the quote applies. It seems witty, but upon further analysis it clearly isn't.

      Delete
  2. We don't have the proper incentives for authors to produce a journal version with full and complete proofs.

    This. An extended abstract in conference proceedings should be considered a preliminary announcement, or to use parallels from other walks of life an intent to file, a preliminary patent application, a memorandum of understanding.

    If the work is not followed up by a properly vetted complete version containing all non-trivial steps, the work should not be considered completed any more than a preliminary patent application is considered a patent.

    Lastly, for the 25% additional contribution, we need to consider towards that figure (1) all additional details and (2) the vetting by referees. So if a paper is pretty much self-contained and straightforward to check, maybe there is no need for a followup journal version. If on the other hand there are 30% more pages with details and examples and a non-trivial amount of refereeing for correctness, then the journal version is measurably different than the conference "looks generally ok" version.

    ReplyDelete
  3. The key features of the PACM proposal include the requirement of at least one revision round where the reviewers get to check that the comments from their reviews have been properly incorporated in the revised papers. That doesn't work with the subtle details of proofs and the short timeline required before a conference - not to mention the large numbers of submissions each PC member has to deal with. There is no way that a PACM version of things could be ready in time for the conferences. In fact, in discussions I have had with those making this proposal, theory is an area where this is considered the least appropriate.

    Over the years, theory conference reviews have gotten much better and more detailed (they originally were not even returned to authors). With our current conference to journal system this effort and information is wasted and as a result journal lead times are longer than they should be. With special issue papers, the editors are PC members who know who the conference reviewers were and can use this information to speed up the process. Other editors may try to guess the PC members who reviewed a paper but will have no way of knowing who outside subreviewers might be. it would be good for the journal reviewing processs did not lose all that valuable work of PC members and subreviewers.

    ReplyDelete
  4. I think, no "hybrid publications" or similar will solve the CORE problem: conference presentations are treated in TCS as publications, not as just announcements of results. I wonder why (some) people are still trying to publish journal versions under this agenda? By wasting their time, and that of referees. (A most annoying aspect when taking into account the time already spent by conference referees.)

    My 2 cents: let us finally separate distinct things - announcements and publications. Let finally make conferences be conferences, meetings, not publishers, archivists.

    ReplyDelete
  5. I wonder why (some) people are still trying to publish journal versions under this agenda?

    Because some people care about the scientific record and they know that in many cases a 10 page abstract falls well short of that. They have the scientific integrity to fill in the gaps rather than taking the convenient way out of not finishing their work and moving on to their next sketch of a result.

    ReplyDelete
  6. We don't have the proper incentives for authors to produce a journal version with full and complete proofs.

    Agreed. However, many conferences do provide strong incentives for authors to produce a version of their paper with full and complete proofs. For example, here is a quote from SODA CFP:

    "The submission must include a full proof of the results, part of which can be placed in the appendix, whose length is not constrained"

    Other conferences, including STOC, FOCS and ICALP, impose similar requirements. As a result, I believe that the (vast) majority of papers published in those conferences have versions with full proofs, available either in proceedings or on arxiv.

    Of course, "full" is quite different from "fully verified". Although determining correctness is an important aspect of reviewing conference submissions, the process is certainly not as thorough as it is for journals. It would be highly beneficial to the community if every paper published in conference proceedings went through a thorough verification process.

    The latter objective is what journals aim to achieve. However, as Paul points out, the separation between conferences and journals significantly increases the total reviewing effort, as many aspects of reviewing are performed twice. As a result, journal editors often find it difficult to secure experts that can review submissions in a timely manner, which prolongs the reviewing process and makes journals a less attractive option overall.

    Personally, I believe that the "journal-integrated" model, where "the conference integrates its committee review process with a journal" (to quote the CACM article) is the best way to proceed. This is, effectively, how journal special issues work, albeit only for small subsets of papers.

    ReplyDelete
  7. @Piotr: I believe that after 10-20 years TCS will finally turn to the essence. No "conference madness", done job is published (not just sketched), etc.

    Current problems with TCS conferences are just "home made".

    ReplyDelete