Here are a few
- Each paper should have at least 3 referee reports.
- Based on available abstracts (but not authors names), PC members should bid for papers. Papers with no bids should be identified early in the process and sent to external referees. Out-of-scope papers should be identified at this early stage and (perhaps?) sent back to the authors right away [this leads to the two-stage reviewing process, which should be another topic for discussion].
- One of the PC members refereeing a paper should be responsible for producing the final report and ensuring that the authors receive proper feedback: with current-size PCs, this load will be roughly 8-10 papers per person, which is very reasonable.
- Each PC member to whom a paper has been assigned should formulate its own opinion on the paper, after reading it. Consulting subreferees should always be considered, rather than remaining at the level of vague impressions on the paper (but this should be done in addition to reading the paper anyway). Referees should not view other referee reviews before submitting theirs: they should read the paper and formulate their own opinions (there is software - such as easychair - to help with this). Submitting a blank report just for the sake of obtaining access to the existing reports should be considered a violation of the honor code. After the initial submission, PC members should see all the reports and either reconsider, or prepare to defend their position during the PC meeting.
- In controversial cases, the paper should be read and commented upon by additional PC members, so that (ideally) a consensus should be reached regarding the paper's value for the conference. The rebuttal phase should help with the resolution of such controversies.
- Consistency in the handling of the papers should be sought. The papers should be judged with the same standards in mind, and without knowledge of the authors' names, affiliation or other scientifically irrelevant information.
- The program committee should have a face-to-face meeting in which each paper is discussed based on its own merits. The identity of the author(s) should not be part of the discussion (hence the need for the double-blind review system). Papers on related topics or using similar techniques may be grouped together during the discussion. This will make it easier to separate significant contributions and novel ideas from incremental work and minor results, and help the committee give proper, content-based feedback to the authors, independent of their status or seniority.
- The PC committee meeting should be held independently of any other conference, workshop or event where some submitting authors may be present and potential conflict-of-interests may arise.
- All reports should be posted on the on-line PC system at least one week before the face-to-face meeting, to give sufficient time to identify star-papers and trouble spots, and to request additional reports.
- No acceptance-rejection should be publicly announced before the rebuttal period. This will add only a few extra days to the refereeing timeline.
- Handling of minor typos and other style issues should be publicized before the submission deadline. The PC should enforce consistency in implementing these criteria, and the consistency should be maintained from year to year and not be left at the discretion of each year's PC chair.
- Handling of special cases: merging of papers, second chances, minor revisions (discussion).
In addition, we need to collect best-practices for handling:
- anonymous submissions: how to identify secondary (non PC) referees [avoiding sending the paper to the authors!!!]
- conflict of interest
Some links to relevant web-accessible publications are collected on the main page of the blog.