[Tccc] ComSoc technical cosponsorship - r...

Henning Schulzrinne hgsatcs.columbia.edu
Thu May 30 14:34:02 EDT 2013



 Also, ranking could be seen as meaning that a TPC member ranks papers within 
their review portfolio (paper #7 is best, #8 second best"). (Infocom tried 
this, I believe.) I don't think that works all that well, but "ranking" may 
well refer to the usual "definite accept" to "definite reject" scale or "in top 
10% of papers". Given the tendency of the first ranking to concentrate around 
the non-committal middle, the latter seems more helpful, but I'm not sure 
that's a "best" practice.  

On May 30, 2013, at 11:40 AM, Ken Calvert <calv... at netlab.uky.edu> wrote:

> Hi Joe - 
> 
> Good idea, thanks for doing this.  I think your proposal is pretty much on 
> target.  Just a couple of thoughts on #6:
> 
>> 6. paper review process            E/A/D
>> 
>>    E = considers average rank AND outlier info, discussion points
>>        also based on natural 'gap' in evaluation
>>    A = considers average rank based on natural gap in evaluation
>>    D = considers rank only
> 
> (i) I interpret these criteria as referring to the accept/reject decision 
> process, rather than the "paper review process".  Perhaps the title should be 
> "acceptance decision process" or something like that?
> 
> (ii) What about considering the transparency of the decision process?
> I.e., whether all (or almost all) decisions are made with in full view of the 
> TPC and with the TPC's approval or at least the opportunity to object.
> 
> (iii) Can you please clarify what you mean by "natural gap in evaluation"? I 
> would probably interpret this to mean that the accept/reject line is drawn, 
> as far as possible, so that there is a clear gap between the (average ratings 
> of) the accepted papers and the rejected papers.  But I don't think that's 
> realistic - especially in large/general conferences, where there are papers 
> from many areas, there will be not be a bright line in the ratings/rankings 
> between rejected and accepted papers.  This also seems to conflict with 
> "considers rank only" being Deficient.  So maybe I've just not understood 
> what this means.
> 
> Cheers,
> 
> KC
> 
> On 29 May 2013, at 14:05 PM, Joe Touch <to... at isi.edu> wrote:
> 
>> Hi, all,
>> 
>> As part of the ComSoc technical cosponsorship (TCS) process, TCs are 
>> supposed to nominate at least two members of the TPC who will monitor 
>> the review process.
>> 
>> However, there doesn't appear to be any guidelines for providing 
>> feedback on that process.
>> 
>> I've drafted the following, which I hope will open a discussion on this 
>> issue. If it evolves into something useful, perhaps it can be posted on 
>> the TC websites for use by those appointed to monitor TC-endorsed TCS'd 
>> meetings.
>> 
>> NB: I've cross-posted this to TCCC, ITC, and TCHSN, which are where I 
>> participate primarily; if any other TC has suggestions, please take the 
>> discussion to the TCCC list if possible.
>> 
>> Thanks,
>> 
>> Joe
>> 
>> -----------------------------------------------------------------
>> 
>> Rating system:
>>     EXCELLENT    best-practice to be aspired to
>>     AVERAGE      acceptable practice
>>     DEFICIENT    cause for concern for ComSoc involvement
>> 
>> 1. TPC participation invitation        E/A/D
>> 
>>    E = before first Call for Papers (CFP) issued
>>    A = before CFP submissions due
>>    D = after CFP submissions due
>> 
>> 2. involvement in CFP promotion        E/A/D
>> 
>>    E = invited to forward CFP and submit
>>    A = invited to submit
>>    D = neither
>> 
>> 3. paper assignment for review        E/A/D
>> 
>>    E = invited to select papers based on expertise and
>>        abstracts/titles
>>    A = invited to select based on topic area
>>    D = not invited to select
>> 
>>    NB: "everyone reviews all" = E
>> 
>> 4. paper review format            E/A/D
>> 
>>    E = includes rank, feedback for author, and private
>>        feedback for TPC discussion
>>    A = includes rank and author feedback
>>    D = includes only rank
>> 
>> 5. TPC meeting                E/A/D
>> 
>>    E = in-person meeting with support for remote
>>    A = in-person with no remote support or only telecon or e-mail
>>    D = no meeting
>> 
>> 6. paper review process            E/A/D
>> 
>>    E = considers average rank AND outlier info, discussion points
>>        also based on natural 'gap' in evaluation
>>    A = considers average rank based on natural gap in evaluation
>>    D = considers rank only
>> 
>> 7. paper reviews returned        E/A/D
>> 
>>    E = >=3 substantive reviews returned with rank and
>>        comments for the authors
>>    A = >=3 substantive reviews returned with rank and
>>        at least a rationale for rejects
>>    D = <3 reviews for some papers, reviews not returned at all,
>>        or only rank provided
>> 
>> 7. paper accept rate            E/A/D
>> 
>>    E = <=50%, based on natural gap in paper evaluation
>>    A = <=50%, not based on 'gap'
>>    D = >50%
>> 
>> ------------------------------------------------------
>> _______________________________________________
>> IEEE Communications Society Tech. Committee on Computer Communications
>> (TCCC) - for discussions on computer networking and communication.
>> Tccc at lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
> 
> Ken Calvert
> Professor and Chair, Computer Science Department
> Acting Director, Vis Center
> University of Kentucky
> 
> 
> 
> 
> 
> _______________________________________________
> IEEE Communications Society Tech. Committee on Computer Communications
> (TCCC) - for discussions on computer networking and communication.
> Tccc at lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
> 


_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
 




More information about the Tccc mailing list