Best Practices

An honor code is implicit in any refereeing process, and best practices should be publicized and used to educate inexperienced reviewers. Anybody who is invited to serve on a Program Committee should be informed from the outset about the expectations, timeline and responsabilities involved. Acceptance of the invitation implies compliance with the published rules.

Here are a few obvious practices for a competitive conference, slightly adapted to the new proposed system of blind-review + rebuttal:

• Each paper should have at least 3 referee reports.
  
  
• Based on available abstracts (but not authors names), PC members should bid for papers. Papers with no bids should be identified early in the process and sent to external referees. Out-of-scope papers should be identified at this early stage and (perhaps?) sent back to the authors right away [this leads to the two-stage reviewing process, which should be another topic for discussion].
  
  
• One of the PC members refereeing a paper should be responsible for producing the final report and ensuring that the authors receive proper feedback: with current-size PCs, this load will be roughly 8-10 papers per person, which is very reasonable.
  
  
• Each PC member to whom a paper has been assigned should formulate its own opinion on the paper, after reading it. Consulting subreferees should always be considered, rather than remaining at the level of vague impressions on the paper (but this should be done in addition to reading the paper anyway). Referees should not view other referee reviews before submitting theirs: they should read the paper and formulate their own opinions (there is software - such as easychair - to help with this). Submitting a blank report just for the sake of obtaining access to the existing reports should be considered a violation of the honor code. After the initial submission, PC members should see all the reports and either reconsider, or prepare to defend their position during the PC meeting.
  
  
• In controversial cases, the paper should be read and commented upon by additional PC members, so that (ideally) a consensus should be reached regarding the paper's value for the conference. The rebuttal phase should help with the resolution of such controversies.
  
  
• Consistency in the handling of the papers should be sought. The papers should be judged with the same standards in mind, and without knowledge of the authors' names, affiliation or other scientifically irrelevant information.
  
  
• The program committee should have a face-to-face meeting in which each paper is discussed based on its own merits. The identity of the author(s) should not be part of the discussion (hence the need for the double-blind review system). Papers on related topics or using similar techniques may be grouped together during the discussion. This will make it easier to separate significant contributions and novel ideas from incremental work and minor results, and help the committee give proper, content-based feedback to the authors, independent of their status or seniority.
  
  
• The PC committee meeting should be held independently of any other conference, workshop or event where some submitting authors may be present and potential conflict-of-interests may arise.
  
  
• All reports should be posted on the on-line PC system at least one week before the face-to-face meeting, to give sufficient time to identify star-papers and trouble spots, and to request additional reports.
  
  
• No acceptance-rejection should be publicly announced before the rebuttal period. This will add only a few extra days to the refereeing timeline.
  
  
• Handling of minor typos and other style issues should be publicized before the submission deadline. The PC should enforce consistency in implementing these criteria, and the consistency should be maintained from year to year and not be left at the discretion of each year's PC chair.
  
  
• Handling of special cases: merging of papers, second chances, minor revisions (discussion).
  
  

In addition, we need to collect best-practices for handling:

• anonymous submissions: how to identify secondary (non PC) referees [avoiding sending the paper to the authors!!!]
  
  
• rebuttals
  
  
• conflict of interest
  

Some links to relevant web-accessible publications are collected on the main page of the blog.

Improving the review system

What should the Computational Geometry community do to improve its review system? Parts of this discussion apply to all its conferences and publications, not just to SoCG.

Topics for discussion:

• Transparency of the SoCG (or any conference's) review process: the authors submitting papers have the right to know how their papers are going to be reviewed, and to receive referee reports meeting at least some agreed-upon minimum quality standards.
  
  
• Checks and balances: double-blind review and a rebuttal period would ensure a much needed balance between the power and the responsabilities of program committee members. The expectation is that the extra effort involved may lead to a more responsible handling of the refereeing duties.
  
  
• Accountability: being asked to serve on a PC (or steering committee) is an honor, and comes with the responsability of accounting for one's judgements and decisions. Currently, there is virtually no mechanism for accountability.
  
  
• Democracy and representability: how is one selected to serve on the PC of a CG conference? Is the whole community represented? Is it appropriate that some conferences rotate the PC duties among basically the same group of people?
  
  
• Educating the referees: inexperienced reviewers should be educated about their duties, and informed on the reviewing process. Experienced reviewers should be reminded of their responsabilities and ethical concerns.