[Tccc] Requesting open feedback to my work (Re: Promoting open on-line research)

Joe Touch touch
Thu Nov 3 13:10:00 EDT 2011



On 11/3/2011 9:48 AM, Michalis Faloutsos wrote:
> As the co-chair of that Global Internet offering, I beg to differ with
> Joe's opinion.
>
> First, it is entirely subjective to claim that the fact that reviews
> were public reduced the number of submissions.

The statistics were that the submissions were significantly down that 
one year, and it took several years to recover to pre-experiment levels.

My claim is that the experiment impacted the number of reviews, as that 
was the only significant change that year, and no other workshop I know 
of experienced a similar change (either at Infocom or elsewhere).

There are many possible reasons why this could occur:

	1- authors didn't want to participate
		we can't know why for sure

	2- reviewers didn't want to participate
		(actually, we know this happened, since a number
		of previous PC members declined to participate
		that year, and checked that the experiment wasn't
		happening the following year before agreeing to
		come back)

	3- the process of conducting an experiment affected other
	aspects of the conference (promotion, etc.)

We can't know for sure what the reason was. What we do know is that it 
did happen.

> As an author, I would love to see who is reviewing my paper. I know a
> lot of people who would.

The question is whether the impact to the community is worth the cost, 
and what you would do with that information if you had it (would it make 
your paper better? would it make your future papers better?)

> One could also attribute this to the existence of mini-papers at
> infocom: still that is not a fact either.

While that's possible, GI has had a different focus from the bulk of 
Infocom papers, and the miniconference from 2007 didn't have papers that 
would have been typical for GI. Further, this impacted only GI at 
Infocom, not the other workshops. That's fairly strong evidence that 
this was GI-specific.

> The trickier part was to get TPC members that were bold enough to accept
> the openness.

Or interested in participating in a poorly designed experiment (no 
control group, in particular).

 > There were no problems doing that though.

That depends on whether you consider the drop in the number of previous 
TPC members a problem, though.

> For more details, some data and outcomes were published here:
> http://ccr.sigcomm.org/online/?q=node/223

Yes, thanks for that link.

Joe



More information about the TCCC mailing list