Topics for discussion:
- Transparency of the SoCG (or any conference's) review process: the authors submitting papers have the right to know how their papers are going to be reviewed, and to receive referee reports meeting at least some agreed-upon minimum quality standards.
- Checks and balances: double-blind review and a rebuttal period would ensure a much needed balance between the power and the responsabilities of program committee members. The expectation is that the extra effort involved may lead to a more responsible handling of the refereeing duties.
- Accountability: being asked to serve on a PC (or steering committee) is an honor, and comes with the responsability of accounting for one's judgements and decisions. Currently, there is virtually no mechanism for accountability.
- Democracy and representability: how is one selected to serve on the PC of a CG conference? Is the whole community represented? Is it appropriate that some conferences rotate the PC duties among basically the same group of people?
- Educating the referees: inexperienced reviewers should be educated about their duties, and informed on the reviewing process. Experienced reviewers should be reminded of their responsabilities and ethical concerns.
17 comments:
Both as an author and as a referee I don't see any point in double-blind reviews and a rebuttal phase as proposed.
As a referee, I want to know who the authors are. I always check the papers I get to referee as carefully as reasonably possible. For that it does not matter who the authors are. However, the identity of the authors does matter when giving feedback.
I am always happy to give extensive comments to a "newby" to help him/her improve the paper. But I do not want to waste my time doing this for experienced authors who should know better, use the referees for proofreading and have a history of ignoring referees' comments and trying with how little care in research and writing they can get away.
On top of that, we know that with conference reviewing we cannot always check the correctness of the results in full detail. In such cases I think it is fair to give the benefit of the doubt more easily to authors with a well-deserved good repetition than to authors who are known to be sloppy.
People with a good or bad reputation usually have that reputation for a reason. (As a matter of fact, I am not easily impressed by big names, but I think it is completely fair when an author who made me referee an erroneous paper before, does not get the benefit of the doubt as easily a second time.) As an author, I would like to have the opportunity to earn myself a reputation that my papers are correct the first time I submit them.
These are my principal objections to double-blind refereeing. In addition I believe it would not work. I got a paper to referee double-blind before. I knew who the authors were immediately, because I knew who was working on this type of stuff, because I got to referee a previous version of the paper for another conference too. Our community is just too small. I think double-blind refereeing would only introduce extra work for both authors and referees for no good reason.
As for a rebuttal phase: as an author I would like it when we would get sent the referee reports before the PC meets, so that we could write a reaction to any referees' comments that are incorrect or unfair. This would increase my confidence that the PC has a fair view of the paper when deciding on acceptance or rejection.
However, I have absolutely no interest in writing a rebuttal *after* the PC meeting. As an author, I have no desire to argue with PC's, and I also would not expect that such a rebuttal phase could change the ranking of papers made up by the PC after long and difficult discussions. I think a rebuttal phase after the PC meeting would just be a waste of time of both PC members and (especially) authors.
about double-blind submissions
- Pretending double-blind is possible in our small field would be cheating people.
The probability of guessing who the authors are is high.
- If we wanted to make it right, we would forbid people
. to talk about their work in workshops like EuroCG or others
. to write a research report and upload it in some archive repository
(these arguments were given by somebody at the socg business meeting).
- Moreover, it would not be possible for authors to give a link to a web page for running some applet, download software or simply a longer version of the paper.
This would have negative impact on the whole process.
- I could have mentioned in introduction that of course double-blind has no sense for SoCG PC members.
When trying to import ideas from other fields, we should first understand them well. In Siggraph for instance, double-blind applies only to tertiary reviewers, and not to 1st and 2nd reviewers who will participate in the discussion, and who must find the tertiary reviewers.
So, the translation to SoCG would be double-blind for sub-reviewers only.
And as mentioned before, I don't believe it would work and I see major drawbacks.
Monique Teillaud
PS: I also posted a few words on the rebuttal in a different part of the blog
- Pretending double-blind is possible in our small field would be cheating people. The probability of guessing who the authors are is high.
In the conferences where double blind has been implemented this has generally not been the case: people are generally wrong about their guesses.
On receiving proper feedback.
Günter Rote
My opinion on this subject is probably extreme. In my opinion, it is the primary job of the committee to make a good selection. Any feedback is voluntary, at the discretion of reviewers, and is not primarily intended to justify the decision. The authors have no right to feedback, beyond whether the paper is accepted or rejected.
Of course, when the decision process is mostly handled electronically, a large amount of comments accumulate anyway, and it would be a waste of effort not to let the authors benefit from it. But currently, there is no standard of how much useful information is given back to authors.
Thus, it may happen that an author just receives a list of typos. Or it may happen that some remarks in the feedback are factually wrong. Or the paper has good grades overall but was nevertheless rejected. The status of the feedback remarks as they are accumulated in some electronic discussion system does not necessarily reflect the final discussion of the committee.
In any case, this feedback is better than no feedback at all; the author is welcome to do with it whatever he or she wants. (grit his or her teeth, or say thank you anyway for the list of typos)
I am aware that such cases make no good impression, and despite explicit disclaimers and remarks, I have seen that authors complained because they believed their paper was rejected because of typos. Sometimes, the PC takes the extra work of "cleansing" the feedback remarks from any nonsense or inconsistency it contains, to make a better impression, or even to extract the main items from the discussion. If they do so, it is nice, of course. But the PC works on a tight schedule, they are exhausted from the discussion after their main job is done.
The authors have no right to feedback, beyond whether the paper is accepted or rejected.
On the contrary, it is our conference. We form the program committee, we submit papers to it, it is organized by a professional society that represent us and we are the audience. As such we "own" it and we have the right to demand pretty much anything.
What we are trying to determine here through public discussion and consensus, is which demands and changes (if any) will help SoCG better achieve its goals, but the right is unquestionably there.
About a rebuttal phase.
Frederic Cazals.
by now, I think the SoCG review process has pros and cons. on one
hand, the final decision tends to be rather collegial, as opposed to
several Computer Graphics conferences where the decision is made by
the PC chairs only, based upon a recommendation written by one of the
reviewers (SGP for example). on the other hand, reviews are sometimes
lapidary ("this is not CG") or even nonsense ("the authors do not seem
to be aware of XXX" while the paper features a full section on topic
XXX.) such reviews partly account for the fact that people not in the
innermost circle are reluctant to submit papers. (I confirmed this
impression by talking to a number of attendees of the recent workshop
on Non linear CG, at the IMA in Minneapolis.)
to improve things, I would advocate a review process with a light
rebuttal phase. the format of a committee consisting of 10-15 people
remains, but the review process consists of 3 rounds:
---first round, approx. 6 weeks: the review phase. primary (committee
members) choose secondary reviewers, and reviews are written. reviews,
together with specific questions for the next phase, are sent to the
authors.
---second round, approx. 1 week: the rebuttal phase. authors prepare a
reply to the questions raised by reviewers. the rebuttal lasts a week,
but, as opposed to Siggraph, is one-shot: reviewers write once,
authors reply once.
benefit: the rebuttal allows authors to answer / clarify important
questions, and avoids low quality reviews (lapidary reviews, paper not
read seriously). the one-shot rebuttal avoids endless and passionate
discussions which are not objective anymore because reviewers are
themselves active on the topic under evaluation. (such endless
discussions currently take place during the Siggraph process.)
---third round, the decision phase. the committee makes the final
decisions, either over the net, or even better, over a physical
meeting.
benefit: having the full committee access all the papers ensures the
decisions are collegial ---and not essentially made by those who
reviewed the paper or the PC chairs only as in a number of
Comp. Graphics conferences.
three things to conclude:
--- all secondary reviewers should get a copy of all the reviews. this
is helpful, especially when such a reviewer wrote a very positive
(negative) review, and the paper got rejected (accepted).
(implemented for Siggraph.)
--- for rejected papers with high variance, a journal submission of
the revised paper accompanied by the SoCG reviews should be made
possible. (also implemented for Siggraph.)
--- finally, one last comment. a light rebuttal phase may not be the
definitive answer, but may be worth a try for a couple of years. the
community should probably run a poll to collect pros (groups that
adopted a rebuttal: Siggraph) and cons (groups that tried and gave up:
are there any!?)
"the authors do not seem to be aware of XXX" while the paper features a full section on topic XXX.
From conversations with colleagues this is happening all too often. The rebuttal phase is meant to address this type of mistakes: factually incorrect statements in the review.
Frederic said
"the community should probably run a poll to collect pros (groups that
adopted a rebuttal: Siggraph) and cons (groups that tried and gave up: are there any!?)"
[ I think this text was already posted in another part of the blog, right?
discussions are already getting hard to follow...
I also wrote another comment on rebuttal somewhere else in the blog]
I did not run a large poll but I asked several people from computer graphics and computer vision.
* To start with, it seems (to be confirmed) that siggraph adopted the rebuttal when the proceedings were transformed into a journal.
This relates with my other remark someweher else in this blog.
* all people (authors, PC members, area chair) I asked answered me that they don't think that their rebuttal does not improve the quality of reviews.
-------
Another remark:
All SoCG PC members can see all papers and all discussions.
And the decisions are made in a collegial way.
This is much more transparent than any other conference mentioned in this blog.
Monique Teillaud
"all secondary reviewers should get a copy of all the reviews. this
is helpful, especially when such a reviewer wrote a very positive
(negative) review, and the paper got rejected (accepted).
(implemented for Siggraph.)"
Are you sure?
I have not seen the other reviews for siggraph. I don't think I was even told whether papers I reviewd were accepted or rejected.
Maybe I was tertiary reviewer only...?
In any case, as I mentioned somewhere else in this blog, all SoCG PC members can see all submitted papers and all reviews and all discussions.
This is much more transparent than what is done for siggraph or other conferences like SGP or computer vision conferences.
Monique Teillaud
I said
Pretending double-blind is possible in our small field would be cheating people. The probability of guessing who the authors are is high.
and Mr/Mrs Y answered
"In the conferences where double blind has been implemented this has generally not been the case: people are generally wrong about their guesses."
Dear Y,
As far as I know, its has not been implemented in what I called our small field.
About other fields, what I heard about Siggraph is that is is often easy to guess, though its is a much larger community than ours.
Monique Teillaud
As far as I know, its has not been implemented in what I called our small field.
Some like to claim that it is easy to guess authorship without even trying, even though actively trained systems only achieve rates of 50%.
Here's a challenge for all those out there claiming authorship can easily be guessed. Go over the list of papers in SoCG 2006 (available here) and tell how you would have correctly guessed authorship. Say had Gunter published anything before on minimum weight triangulation? Was David Mount's previous paper on the range searching six years before a dead give away? Exactly which paper from Jack foretold his result inscribing spheres in polytopes? Sure, Micha has done a lot of work on shortest paths on convex polytopes, but so have Subash, Sariel, John H. and Pankaj (as well as combinations thereof). So how exactly could you tell that this paper was by Micha and not by Micha and Sariel, or by Sariel alone, though perhaps following up on ideas of Micha?
Double-blind or not: practical issues
by Günter Rothe
There is the issue whether double-blind is really desirable, supposedly because it is more
objective, or whether the down-sides outweigh the benefits. On the other side, there are down-to-earth practical issues of implementing it. Let me dwell on those.
SIGGRAPH, as I understand, has a double-blind reviewing system where they take serious measures to ensure anonymity: you are not allowed to give a seminar talk about your submitted work when a PC member might be present, you must not post your work on your web pages, and so on.
I think such a system would not be compatible with our culture, with CCCG, EWCG, Fall Worskhop, Dagstuhl workshops, preprints on web pages or in arXiv.
Also, SICGRAPH is a much larger circus than ours. My impression is that they have a very
streamlined and formalized process, which solves, for example, the issue of supplementary material (videos and perhaps applets), which would not be easy to establish for SoCG. (Any help is appreciated.) It is the graphics community's central event. People live and die for SIGACT. That is why it is central to them, and that is also why authors and reviewers are more willing to submit to a tight schedule, and authors can be expected to respond to the rebuttal remarks in short time, and so on.
(According to an article on the history of SIGGRAPH, in the May 2007 issue of Communications of the ACM, "SIGGRAPH was the place to publish in computer graphics, and one of the hallmarks of the conference was the feeling that publishing one SIGGRAPH paper could be enough for an academic person to receive tenure."
In our case (as opposed to SIGGRAPH), we need not commit suicide if our great SoCG submission is rejected, we can always submit it as a full paper to a journal, which allows
for extensive refereeing under no time pressure.)
Back to double-blind.
What would be relatively easy to implement is some sort of "double-blind-light".
Authors would have to submit anonymously, but other than carefully phrasing references in the paper in order not to reveil their identity, would not be required to take any other measures. PC members and reviewers might still search the web to find out who the authors are, if they are "malicious", but of course would be discouraged to do so, (and they don't have much time anyway.
If the authors' names are unknown to PC members, it would be necessary to establish some sort of coordination for avoiding sending a paper to the author or a close colleage for review. The chair might be required to handle this. (But of course, if a PC member proposes a reviewer and the chair rejects it, then the PC member gets a clue about the authorship.) There are conferences where the chair does no reviewing of his/her own, but just these coordination tasks.
Another possibility, which can be implemented without much ado, as an experiment, is "double-blind-very-light": The authorship is known to PC members. They know whom they cannot ask for subreviews, but the subreviewers don't see the authors' names.
If I (as a reviewer) happen to know (or believe to guess) the identity of the authors, then so be it. What I have read about experiences in other communities on the linked web
pages of this forum is that people were not always right when they believed they could guess the authors. Even if there is no measurable outcome, it would at least be an interesting psychological self-experiment for each reviewer.
(One might offer a bonus for guessing the authors correctly without illicit means like searching the web for preprints. This would add to the joy of reviewing.)
(BTW, I had published before about minimum-weight triangulations.
I am in favor a double-blind review process (fair) without a rebuttal phase (reduce the workload).
In essence, this makes it difficult for an author to post his/her results on the web while the paper is under review. But this certainly makes the review process more fair.
Guessing the authorship for papers is hard; yes, you might be able to follow the writing style to guess it right for some paper, but given 20 papers it will be hard for anybody to guess more than 30-40% correctly.
Double-blind or not: principal issues
Let me list some down-sides of bouble-blind reviewing.
(Some of these are less severe when the identity of reviewers is known to the PC.)
1) It would not be possible to contact authors directly by e-mail if something is unclear. Such contacts would have to become part of a formalized "rebuttal" process (in case such a process is installed).
(I have had occasions where reviewers wanted to guard their anonymity, and I have passed back and forth questions and answers in my capacity as a PC member, but this is rather time-consuming; in principle, it would be no problem for an electronic discussion support system to manage such an exchange anonymously.)
Currently, such contacts are at the discretion of the reviewers, who are willing to spend time on e-mail exchanges, perhaps hoping to save time in not having to think about some question on their own. (I have had mixed experiences in such cases; some people have withdrawn the paper right away, after considering my question seriously with their coauthors and not finding an answer within one or two days; others have tried to evade the issue and argue endlessly around it.)
I have been in PCs where it was the strict policy (in the spirit of fairness) that papers should be evaluated based on the submitted version alone. If you know a tech-report or have heard a talk about the paper, possibly with further improvements or even stronger results, you should ignore this knowledge.
My "official" position (this is what I tell authors, in case they dare to complain) is that papers are reviewed exactly as submitted. Authors should prepare their submission carefully: Proofs should of course be 100% correct, with well-chosen examples and figures illustrating the matter. Any accidental omissions, glitches, bad writing, misunderstandings, failure to list important possible applications in the introduction or to mention important related literature, are entirely the author's fault and put the paper at the mercy of the one committee member who likes the paper nevertheless, and is willing to spend the extra time to dig through the paper or explore how it is related to or differs from other papers, or perhaps have a back-and-forth discussion over e-mail.
(On the other hand, each paper deserves a sincere effort on the part of the reviewers. (This is what I tell reviewers.) Even if it is hard to read but purports to contain an important idea or result, maybe one person can sacrifice a day to dig into it.
I am getting carried away. Back to double-blind.
2) It would not be possible to perform "affirmative action".
A good but not great paper might more likely be accepted if it is by a student, a newcomer or practitioner or by an "outsider" from a related field, or from an under-represented region of the world, than if it is the n+1 st paper by a most noble well-known well-respected person of our field. Bad language is more likely to be forgiven if it comes from a non-English-speaking "newby" than if one of the authors is known to be fluent in English but apparently did not have the time to read the submission. Similar thoughts have been mentioned by herman in the very first comment in this blog.
Another question on the list of evaluation criteria that is often discussed is: Would it be a good talk? Would it make it worth while for people to attend and enjoy the conference, as opposed to skipping a session in favor of sightseeing or meeting some friends? Of course this includes the question of the person who is likely to present the talk. People who notoriously give bad talks without putting any effort in preparation might get a malus in the evaluation. (But I can't think of any case where this would apply.)
Of course these questions are only secondary criteria, which may aid a decision when a paper is near the edge.
It is debatable whether such "affirmative action" or other sorts of political or moral decisions are fair and desirable, or whether the decision should be based on "contents" alone.
3) It would be less easy to track very incremental (or even duplicate) work if the authors choose not to cite their own closely related work. (This is currently hardly an issue at all.)
--
[ In the spirit of bubble-blind, I am post this anymousy. See if can guess who you am.
"affirmative action" tends to affirm the status quo, it is what you claim when your candidate is not good enough, but it is clearly _your_ candidate. If objectivity could be had, we would be better served with objectivity than with the politics of affirmative action. I have a strong ethical objection against any referee who needs to know the identity of the author to decide on his judgement on a paper.
I do not believe objectivity is to be had.
Truth is, that the quality of the referee's reports in such a conference will necessarily be lower than in a good journal, and more colored by group interests and `do I know that guy?'.
What, e.g., Eli Goodman does for DCG just cannot be achieved by a program committee, most of whose members do this just once, in a hurry, and with a limited knowledge of subjects, people, and interests outside their immediate group. To get a good, just and helpful referee's report, you need to find a referee with the same interests, and know his prejudices and pet peeves.
The referee should actually be interested to make the manuscript a good paper, and make the best out of it.
No chance for a program committee to reach that level.
So we have to live with realistic expectations, colored by all kinds of influences and prejudices. I think what should be done, instead of creating a pseudo-objectivity of a more complicated review system, is to admit the subjectivity of these decisions, and advise anyone with a good paper to go to a journal instead.
My personal experience: I clearly do not belong to the SoCG community, my acceptance rate there is next to 0, whereas for journals it is 100%. I state this without complaining, I never complain about an editor's decision,
but I offer this as evidence that SoCG acceptance is not the same as `the best papers on topics related to computational geometry'.
The SoCG is a showcase of a group, and no amount of complication of the review system will change that.
Peter Brass
Peter Brass said: `affirmative action affirms the status quo'.
There must be a fundamental misunderstanding about what I wrote about "affirmative action". The mentioned examples are clearly the opposite of "my candidate" or "the guy I know", but rather the people I wouldn't normally know. On the contrary, without some positive effort, there is a natural tendency to conserve the status quo, of the conference to become the "showcase of a group". Counteracting this natural tendency is what I had in mind when speaking about affirmative action. One may object to such considerations on principal grounds, but there is nothing unethical in these intentions.
--
Günter Rote
Peter Brass said: `the *best* papers on topics related to computational geometry'.
I think this is an oversimplification of what SoCG tries to achieve. Besides pure "quality", there are many other goals, like a balance of subjects, appropriateness for a conference, potential to stimulate discussion, etc. `The most interesting papers on topics related to computational geometry' is a better characterization of what SoCG aims to be.
It is perfectly o.k. for a paper that is 100% appropriate as a journal publication to be rejected from SoCG. On the other hand, I envision papers that would be clearly be rejected from a journal but be great for SoCG, see David Patterson's article "The health of research conferences and the dearth of big idea papers" from the December 2004 issue of the Communications of the ACM, which is linked from the pages of this blog. I must admit we don't see such papers much. (And would they be accepted?)
--
Günter Rote
Post a Comment