[Tccc] V5 - ComSoc technical cosponso...
Laura Marie Feeney
lmfeeneyatsics.se
Thu May 30 19:47:43 EDT 2013
Hi Joe,
Rather than more guidelines, I would prefer to see (from IEEE and
others) a stronger commitment to data-driven understanding of the review
process and support for (semi-) controlled experiments on improving it.
I'd bet that EDAS knows things about the review process that we don't
know ourselves...
Serious work probably requires better consensus about data handling and
sharing than we have now, especially if we want to correlate over
multiple conferences. But I think it would be helpful to begin this
effort and also to raise expectations on Chairs for scientific approach
to the problem.
For specific comments on the guidelines:
> 2. involvement in CFP promotion E/A/D
It's not clear that this a significant indicator of quality. (At least,
I'm not lacking CFP spam.)
> 3. paper assignment for review E/A/D
What is best practice?: Even with double blinding, giving reviewers too
much role in selection can be vulnerable to collusion or (more likely)
just inbreeding among circles of people who have similar ideas about
what's interesting. Is there data? Do papers tend to have correlated
sets of reviewers and does it affect variability in review scores? If
correlation exists, does it persist across conferences? Can assignment
algorithms be designed to mitigate this?
> 5. TPC meeting E/A/D
Disagree. TPC meetings are expensive, environmentally unfriendly, and
reduce TPC diversity. Phone-only meetings are better, but are still
unavoidably at 2am in either America, Europe or Asia.
> 6. paper review process E/A/D
>
> E = considers average rank AND outlier info, discussion points
> also based on natural 'gap' in evaluation
> A = considers average rank based on natural gap in evaluation
> D = considers rank only
I'm somewhat confused about the idea of 'natural gap'. Conventional
wisdom is that most conferences have a few clear accepts, many clear
rejects, and a certain amount of randomness (along with careful
evaluation, of course) in the middle.
N) I was quite surprised that rules for delegating reviews weren't
considered relevant. I was slightly surprised that review load wasn't
considered important.
N) I was disappointed that reviewer diversity (gender/national
/institutional/year-on-year turnover) wasn't considered relevant.
Regards,
Laura
_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
More information about the Tccc
mailing list