Here are a few
- Each paper should have at least 3 referee reports.
- Based on available abstracts (but not authors names), PC members should bid for papers. Papers with no bids should be identified early in the process and sent to external referees. Out-of-scope papers should be identified at this early stage and (perhaps?) sent back to the authors right away [this leads to the two-stage reviewing process, which should be another topic for discussion].
- One of the PC members refereeing a paper should be responsible for producing the final report and ensuring that the authors receive proper feedback: with current-size PCs, this load will be roughly 8-10 papers per person, which is very reasonable.
- Each PC member to whom a paper has been assigned should formulate its own opinion on the paper, after reading it. Consulting subreferees should always be considered, rather than remaining at the level of vague impressions on the paper (but this should be done in addition to reading the paper anyway). Referees should not view other referee reviews before submitting theirs: they should read the paper and formulate their own opinions (there is software - such as easychair - to help with this). Submitting a blank report just for the sake of obtaining access to the existing reports should be considered a violation of the honor code. After the initial submission, PC members should see all the reports and either reconsider, or prepare to defend their position during the PC meeting.
- In controversial cases, the paper should be read and commented upon by additional PC members, so that (ideally) a consensus should be reached regarding the paper's value for the conference. The rebuttal phase should help with the resolution of such controversies.
- Consistency in the handling of the papers should be sought. The papers should be judged with the same standards in mind, and without knowledge of the authors' names, affiliation or other scientifically irrelevant information.
- The program committee should have a face-to-face meeting in which each paper is discussed based on its own merits. The identity of the author(s) should not be part of the discussion (hence the need for the double-blind review system). Papers on related topics or using similar techniques may be grouped together during the discussion. This will make it easier to separate significant contributions and novel ideas from incremental work and minor results, and help the committee give proper, content-based feedback to the authors, independent of their status or seniority.
- The PC committee meeting should be held independently of any other conference, workshop or event where some submitting authors may be present and potential conflict-of-interests may arise.
- All reports should be posted on the on-line PC system at least one week before the face-to-face meeting, to give sufficient time to identify star-papers and trouble spots, and to request additional reports.
- No acceptance-rejection should be publicly announced before the rebuttal period. This will add only a few extra days to the refereeing timeline.
- Handling of minor typos and other style issues should be publicized before the submission deadline. The PC should enforce consistency in implementing these criteria, and the consistency should be maintained from year to year and not be left at the discretion of each year's PC chair.
- Handling of special cases: merging of papers, second chances, minor revisions (discussion).
In addition, we need to collect best-practices for handling:
- anonymous submissions: how to identify secondary (non PC) referees [avoiding sending the paper to the authors!!!]
- rebuttals
- conflict of interest
Some links to relevant web-accessible publications are collected on the main page of the blog.
9 comments:
A rebuttal phase for the SoCG review process?
by now, I think the SoCG review process has pros and cons. on one
hand, the final decision tends to be rather collegial, as opposed to
several Computer Graphics conferences where the decision is made by
the PC chairs only, based upon a recommendation written by one of the
reviewers (SGP for example). on the other hand, reviews are sometimes
lapidary ("this is not CG") or even nonsense ("the authors do not seem
to be aware of XXX" while the paper features a full section on topic
XXX.) such reviews partly account for the fact that people not in the
innermost circle are reluctant to submit papers.
to improve things, I would advocate a review process with a light
rebuttal phase. the format of a committee consisting of 10-15 people
remains, but the review process consists of 3 rounds:
---first round, approx. 6 weeks: the review phase. primary (committee
members) choose secondary reviewers, and reviews are written. reviews,
together with specific questions for the next phase, are sent to the
authors.
---second round, approx. 1 week: the rebuttal phase. authors prepare a
reply to the questions raised by reviewers. the rebuttal lasts a week,
but, as opposed to Siggraph, is one-shot: reviewers write once,
authors reply once.
benefit: the rebuttal allows authors to answer / clarify important
questions, and avoids low quality reviews (lapidary reviews, paper not
read seriously). the one-shot rebuttal avoids endless and passionate
discussions which are not objective anymore because reviewers are
themselves active on the topic under evaluation. (such endless
discussions currently take place during the Siggraph process.)
---third round, the decision phase. the committee makes the final
decisions, either over the net, or even better, over a physical
meeting.
benefit: having the full committee access all the papers ensures the
decisions are collegial ---and not essentially made by those who
reviewed the paper or the PC chairs only as in a number of
Comp. Graphics conferences.
three things to conclude:
--- all secondary reviewers should get a copy of all the reviews. this
is helpful, especially when such a reviewer wrote a very positive
(negative) review, and the paper got rejected (accepted).
(implemented for Siggraph.)
--- for rejected papers with high variance, a journal submission of
the revised paper accompanied by the SoCG reviews should be made
possible. (also implemented for Siggraph.)
--- finally, one last comment. a light rebuttal phase may not be the
definitive answer, but may be worth a try for a couple of years.
journal submission of the revised paper accompanied by the SoCG reviews?
Can you explain this? A journal submission is always possible, whether the paper is rejected or accepted, and even encouraged, in the latter case.
Nobody can prevent the author from attaching reviews he/she received (provided they were made accessible).
-- Günter Rote
best practices (2) "rules and guidelines"
Ileana's proposal of "best practices" contains a mixture of three different things:
(1) fundamental issues, like the double-blind submission, rebuttals, or whether the PC should meet face-to-face
(2) lot of detailed "rules", like that the review should be ready one week before the meeting and such details, or how the bidding is organized.
(3) and then on another level ethical issues like an "honor code".
Let me remark on the "rules".
These are certainly good guidelines and suggestions. Those details, like the setup of the schedule and the choice of an electronic PC support system, are currently left to the discretion of the PC (chairs), and they should remain so. (Why one week and not five days?)
It is indeed worth while to gather the experience that PCs have had with different procedures, or in particular with different electronic PC systems, list the pro's and con's, for the benefit of future PC's. What went wrong and what turned out to be a good choice? This is not so much a question of opinions, but of experience and information. "Best practices" could develop from such a collected experience in the end. I hope that such a collection will develop on this discussion forum. (Or is there a better medium somewhere else?)
Even the question of a physical versus electronic-only meeting, and colocating it with another meeting is a matter of practicality (and money) and not a question of right-and-wrong.
--
Günter Rote
P.S. I don't understand what the issue with handling of minor typos is.
(They should be corrected, of course.)
Frederic said:
"the rebuttal allows authors to answer / clarify important
questions, and avoids low quality reviews (lapidary reviews, paper not read seriously)."
3 things:
- I participated in two SoCG PCs so far. I noticed how seriously PC members were working.
Still, it is always possible that one PC member does not read one of his 45 papers carefully enough. But then, there are 2 or 3 other PC members also in charge of it, and statistically, they will not all read the paper badly. And they will participate in discussions.
The final decision is not made on the basis of the bad review of 1 particular PC member.
I think that trying to work even more than usually done by PCs on the feedback given to authors is likely to avoid the wrong impression that a paper was rejected because one single person did not read it well.
Or maybe sending no feedback at all would also be a solution to this ;)
- When a review looks stupid, it may be because the reader could not read the paper because it was not enough well written.
Authors should accept to share the responsibility when their paper is rejected, it is not always only a bad PC's fault...
- In the same direction, if an author needs a rebuttal to clarify his paper, then it is probably because the paper is not well written. He can convince people during the rebuttal phase that his results are good. So what? in a journal paper, the reviewers would ask for revisions, and read the revised version, and maybe ask revisions again, and after a few versions the paper would be accepted.
SoCG is a conference, the PC sees only the submitted version. If the paper needs to be rewritten, it can be submitted again one year later.
Monique Teillaud
Guenter said:
"Ileana's proposal of "best practices" contains a mixture of three different things:"
I agree.
Some of them are personal opinions, whereas they are presented as THE Truth. This sounds really strange to me.
In particular, I don't agree with "The program committee should have a face-to-face meeting".
There was no face-to-face meeting in 2005 and it was really fine that way.
In a face-to-face meeting, differences between people may induce strong biases.
Some people speak faster, some people speak louder, some people are more fluent in English, and so on. Still, their opinion on a paper is not necessarily more clever than the opinion of a shy and slower person.
Also, a face-to-face meeting has a very limited duration, nobody can afford to read a paper again when there is a difficult discussion on it.
In an electronic meeting (typically 3 weeks long), these two problems disappear. People have time to think before reacting and talking, and they have time to read a paper again.
Even more, PC members whom a given paper was not assigned to, can also read it if they think they can add something useful to the discussion.
Monique Teillaud
The final decision is not made on the basis of the bad review of 1 particular PC member.
This is just not so. If a paper gets two (correct) accepts but with a low confidence vote and a third (incorrect) reviewer gives a strong reject but with a high confidence it gets rejected.
If the incorrect reviewer had given an accept the paper would have gotten in.
Sure, I'm willing to admit that in many cases of "bad reviewers" are in truth just the consequence of bad writing, but by the same token Monique must be willing to admit that many bad reviews are borne of a lack of time (I've been in enough PCs myself to know that this is very much the case).
Re: This is just not so. If a paper gets two (correct) accepts but with a low confidence vote and a third (incorrect) reviewer gives a strong reject but with a high confidence it gets rejected.
This is not my experience. Instead, I think it is more accurate to say that if a paper gets two accepts and one strong reject (regardless of confidence) then what it gets is more attention from the PC, both the ones already involved and others who see the controversy and want to understand what it's about.
anonymous said
"If a paper gets two (correct) accepts but with a low confidence vote and a third (incorrect) reviewer gives a strong reject but with a high confidence it gets rejected."
Hi Anonymous
No, decisions are not made mechanically by computing the weighted average of scores...
There are discussions, that can be quite long on some papers. Arguments must be given to convince the other PC members. Giving a low score with high confidence but no good argument will not help to reject a paper.
Monique Teillaud
"Monique must be willing to admit that many bad reviews are borne of a lack of time (I've been in enough PCs myself to know that this is very much the case)."
Dear Mr/Mrs X,
I already admitted this in a previous comment.
But I added that when 3 or 4 people in total review a paper, then the paper is still well read, even if one review is bad.
And then, the bad review does not make the decision, even if the associated confidence rate is high.
Other reviewers, who read the paper correctly, will read the bad review and will not follow wrong arguments.
Especially if the discussion is electronic, then the bad confident reviewer will not shout louder than others.
Monique Teillaud
Post a Comment