[Tccc] V5 - ComSoc technical cosponso...

Laura Marie Feeney lmfeeneyatsics.se
Thu May 30 19:47:43 EDT 2013



 Hi Joe,

Rather than more guidelines, I would prefer to see (from IEEE and 
others) a stronger commitment to data-driven understanding of the review 
process and support for (semi-) controlled experiments on improving it.  

I'd bet that EDAS knows things about the review process that we don't 
know ourselves...

Serious work probably requires better consensus about data handling and 
sharing than we have now, especially if we want to correlate over 
multiple conferences.  But I think it would be helpful to begin this 
effort and also to raise expectations on Chairs for scientific approach 
to the problem.

For specific comments on the guidelines:

> 2. involvement in CFP promotion        E/A/D

It's not clear that this a significant indicator of quality. (At least, 
I'm not lacking CFP spam.)

> 3. paper assignment for review        E/A/D

What is best practice?: Even with double blinding, giving reviewers too 
much role in selection can be vulnerable to collusion or (more likely) 
just inbreeding among circles of people who have similar ideas about 
what's interesting.  Is there data?  Do papers tend to have correlated 
sets of reviewers and does it affect variability in review scores?  If 
correlation exists, does it persist across conferences?  Can assignment 
algorithms be designed to mitigate this?

> 5. TPC meeting                E/A/D

Disagree. TPC meetings are expensive, environmentally unfriendly, and 
reduce TPC diversity. Phone-only meetings are better, but are still 
unavoidably at 2am in either America, Europe or Asia.

> 6. paper review process            E/A/D
>
>      E = considers average rank AND outlier info, discussion points
>          also based on natural 'gap' in evaluation
>      A = considers average rank based on natural gap in evaluation
>      D = considers rank only

I'm somewhat confused about the idea of 'natural gap'.  Conventional 
wisdom is that most conferences have a few clear accepts, many clear 
rejects, and a certain amount of randomness (along with careful 
evaluation, of course) in the middle.

N) I was quite surprised that rules for delegating reviews weren't 
considered relevant. I was slightly surprised that review load wasn't 
considered important.

N) I was disappointed that reviewer diversity (gender/national 
/institutional/year-on-year turnover) wasn't considered relevant.

Regards,
Laura

_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
 




More information about the TCCC mailing list