[Tccc] Requesting open feedback to my work (Re: Promoting open on-line research)

Joe Touch touch
Thu Nov 3 11:44:55 EDT 2011



On 11/3/2011 7:00 AM, Pars Mutaf wrote:
...
> The reviews should be publicly available to everyone.

There have been attempts to explore this and other models, e.g., in no 
particular order:

	A- author rebuttal of reviews
	B- blind reviews
	C- double-blind process
		where the paper authors are hidden during review
	D- public reviews
		where reviews are published with the paper
	E- open reviews
		where the author sees the reviewer's names
	F- adding a venue for papers on the 'borderline' of the
	   main conference

Speaking as someone who has participated as a PC member in these in 
various places (as an individual, not as TCCC Chair):

A was tried at Infocom (and elsewhere). The goal was to avoid a paper 
being discarded because of an incorrect review. The result was a 
substantial increase in review time (actually, it ended up resulting in 
less time for reviewers to complete their reviews due to a fixed yearly 
cycle), but no substantial change in paper handling. Most of the 
rebuttals did not point out review errors, but rather disagreed with 
review opinion.

B is currently typical.

C is used at Sigcomm and more recently at ICNP. It is intended to avoid 
favoritism, but IMO it also tends to work against systems work that has 
been vetted in workshops and symposia in parts.

D has been tried for some CCR papers, where a single review or summary 
of the reviews is presented.

E was tried at Global Internet a number of years ago, and nearly killed 
the meeting. Submissions went down over 50%. The result was much more 
pleasantly-written reviews, but the reviews were (IMO) less useful.

F was introduced at Infocom several years ago. IMO, it simply introduced 
a second borderline, and made it very difficult to distinguish between 
full accepts and "consolation prize" accepts.

All of the above were introduced to address a perceived or real concern. 
None of them was tested in a true experiment (e.g., with a control group 
during the same year). Most of them (IMO) were introduced because chairs 
believe that mechanism can address review process problems. IMO, there 
is only one good solution for all such problems:

	PC chairs MUST review the reviews. EVERY review. EVERY year.
		Reviews whose ranks are not substantiated by
		meaningful comment must be both discarded and
		replaced.

Overall, IMO, it is useful to understand that:

	- reviewing is an imperfect process

	- a paper's quality is determined by what the reader
	  receives (goodput), not what is sent (offered load) ;-)

	- papers are rejected because of the lack of positive comments,
	  not for any single negative comment
		(so arguing each negative comment in a review
		won't fix a paper - many reviewers simply provide
		sufficient negatives to justify a decision, but
		could provide other negatives if asked)

	- at large conferences, papers are rejected after substantial
	  decision
		e.g., at Infocom, a paper is either a unanimous reject
		by three reviewers, OR is then considered by at least
		an additional 8-10 people during the PC meeting

I see none of these changing in an open process.

Joe





More information about the TCCC mailing list