[Tccc] ComSoc technical cosponsorship - r...
Anthony Ephremides
etonyatumd.edu
Thu May 30 17:10:32 EDT 2013
I think this whole idea is another example of bureaucratic thinking within the
Society. Any such ranking will be fraught with inaccuracies and will convey
erroneous messages. We all know a good conference when we see one. Rejection
rates as a metric of quality?? Holding a TPC meeting as a measure of quality?
Who will measure the quality of the reviews and of the authors?
My suggestion is to scrap the project.
AE
Anthony Ephremides
Distinguished University Professor and
Cynthia Kim Eminent Professor of
Information Technology
ECE dept and ISR
University of Maryland
College Park, MD 20742
301-405-3641
etony(at)umd(dot) edu
-----Original Message-----
From: Henning Schulzrinne [mailto:h... at cs.columbia.edu]
Sent: Thursday, May 30, 2013 2:33 PM
To: Ken Calvert
Cc: Tccc at lists.cs.columbia.edu; tc... at ri.uni-tuebingen.de; i... at comsoc.org; Joe
Touch
Subject: Re: [Tccc] ComSoc technical cosponsorship - rating the review process
Also, ranking could be seen as meaning that a TPC member ranks papers within
their review portfolio (paper #7 is best, #8 second best"). (Infocom tried
this, I believe.) I don't think that works all that well, but "ranking" may
well refer to the usual "definite accept" to "definite reject" scale or "in top
10% of papers". Given the tendency of the first ranking to concentrate around
the non-committal middle, the latter seems more helpful, but I'm not sure
that's a "best" practice.
On May 30, 2013, at 11:40 AM, Ken Calvert <calv... at netlab.uky.edu> wrote:
> Hi Joe -
>
> Good idea, thanks for doing this. I think your proposal is pretty much on
> target. Just a couple of thoughts on #6:
>
>> 6. paper review process E/A/D
>>
>> E = considers average rank AND outlier info, discussion points
>> also based on natural 'gap' in evaluation
>> A = considers average rank based on natural gap in evaluation
>> D = considers rank only
>
> (i) I interpret these criteria as referring to the accept/reject decision
> process, rather than the "paper review process". Perhaps the title should be
> "acceptance decision process" or something like that?
>
> (ii) What about considering the transparency of the decision process?
> I.e., whether all (or almost all) decisions are made with in full view of the
> TPC and with the TPC's approval or at least the opportunity to object.
>
> (iii) Can you please clarify what you mean by "natural gap in evaluation"? I
> would probably interpret this to mean that the accept/reject line is drawn,
> as far as possible, so that there is a clear gap between the (average ratings
> of) the accepted papers and the rejected papers. But I don't think that's
> realistic - especially in large/general conferences, where there are papers
> from many areas, there will be not be a bright line in the ratings/rankings
> between rejected and accepted papers. This also seems to conflict with
> "considers rank only" being Deficient. So maybe I've just not understood
> what this means.
>
> Cheers,
>
> KC
>
> On 29 May 2013, at 14:05 PM, Joe Touch <to... at isi.edu> wrote:
>
>> Hi, all,
>>
>> As part of the ComSoc technical cosponsorship (TCS) process, TCs are
>> supposed to nominate at least two members of the TPC who will monitor
>> the review process.
>>
>> However, there doesn't appear to be any guidelines for providing
>> feedback on that process.
>>
>> I've drafted the following, which I hope will open a discussion on
>> this issue. If it evolves into something useful, perhaps it can be
>> posted on the TC websites for use by those appointed to monitor
>> TC-endorsed TCS'd meetings.
>>
>> NB: I've cross-posted this to TCCC, ITC, and TCHSN, which are where I
>> participate primarily; if any other TC has suggestions, please take
>> the discussion to the TCCC list if possible.
>>
>> Thanks,
>>
>> Joe
>>
>> -----------------------------------------------------------------
>>
>> Rating system:
>> EXCELLENT best-practice to be aspired to
>> AVERAGE acceptable practice
>> DEFICIENT cause for concern for ComSoc involvement
>>
>> 1. TPC participation invitation E/A/D
>>
>> E = before first Call for Papers (CFP) issued
>> A = before CFP submissions due
>> D = after CFP submissions due
>>
>> 2. involvement in CFP promotion E/A/D
>>
>> E = invited to forward CFP and submit
>> A = invited to submit
>> D = neither
>>
>> 3. paper assignment for review E/A/D
>>
>> E = invited to select papers based on expertise and
>> abstracts/titles
>> A = invited to select based on topic area
>> D = not invited to select
>>
>> NB: "everyone reviews all" = E
>>
>> 4. paper review format E/A/D
>>
>> E = includes rank, feedback for author, and private
>> feedback for TPC discussion
>> A = includes rank and author feedback
>> D = includes only rank
>>
>> 5. TPC meeting E/A/D
>>
>> E = in-person meeting with support for remote
>> A = in-person with no remote support or only telecon or e-mail
>> D = no meeting
>>
>> 6. paper review process E/A/D
>>
>> E = considers average rank AND outlier info, discussion points
>> also based on natural 'gap' in evaluation
>> A = considers average rank based on natural gap in evaluation
>> D = considers rank only
>>
>> 7. paper reviews returned E/A/D
>>
>> E = >=3 substantive reviews returned with rank and
>> comments for the authors
>> A = >=3 substantive reviews returned with rank and
>> at least a rationale for rejects
>> D = <3 reviews for some papers, reviews not returned at all,
>> or only rank provided
>>
>> 7. paper accept rate E/A/D
>>
>> E = <=50%, based on natural gap in paper evaluation
>> A = <=50%, not based on 'gap'
>> D = >50%
>>
>> ------------------------------------------------------
>> _______________________________________________
>> IEEE Communications Society Tech. Committee on Computer
>> Communications
>> (TCCC) - for discussions on computer networking and communication.
>> Tccc at lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>
> Ken Calvert
> Professor and Chair, Computer Science Department Acting Director, Vis
> Center University of Kentucky
>
>
>
>
>
> _______________________________________________
> IEEE Communications Society Tech. Committee on Computer Communications
> (TCCC) - for discussions on computer networking and communication.
> Tccc at lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>
_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
More information about the TCCC
mailing list