[Tccc] ComSoc technical cosponsorship - r...

Joe Touch touchatisi.edu
Thu May 30 18:08:19 EDT 2013



 Hi, Anthony,

The point of this is to support a bureaucratic process - TCS, and TC 
endorsement. Right now, endorsement requires naming two TC members who 
will oversee a meeting. There is no suggestion to what kinds of 
information they will report back to the TC.  

This isn't intended as a fixed form. TC members can report anything they 
want. TCs can ask anything they want. This is just intended as a place 
to start.

I appreciate that others on this list have various views of whether this 
can or should be a TC or ComSoc doc. I'll leave that to the TC chairs or 
the ComSoc to decide.

Regardless of whether they decide to move forward with this or "scrap" 
it, I will be posting it on a site I maintain for conference best 
practices anyway, FWIW.

Joe

On 5/30/2013 2:09 PM, Anthony Ephremides wrote:
> I think this whole idea is another example of bureaucratic thinking within 
> the Society. Any such ranking will be fraught with inaccuracies and will 
> convey erroneous messages. We all know a good conference when we see one. 
> Rejection rates as a metric of quality?? Holding a TPC meeting as a measure 
> of quality? Who will measure the quality of the reviews and of the authors?
> My suggestion is to scrap the project.
>
> AE
>
>
>
>
>
> Anthony Ephremides
> Distinguished University Professor and
> Cynthia Kim Eminent Professor of
> Information Technology
> ECE dept and ISR
> University of Maryland
> College Park, MD 20742
> 301-405-3641
> etony(at)umd(dot) edu
>
>
> -----Original Message-----
> From: Henning Schulzrinne [mailto:h... at cs.columbia.edu]
> Sent: Thursday, May 30, 2013 2:33 PM
> To: Ken Calvert
> Cc: Tccc at lists.cs.columbia.edu; tc... at ri.uni-tuebingen.de; i... at comsoc.org; 
> Joe Touch
> Subject: Re: [Tccc] ComSoc technical cosponsorship - rating the review process
>
> Also, ranking could be seen as meaning that a TPC member ranks papers within 
> their review portfolio (paper #7 is best, #8 second best"). (Infocom tried 
> this, I believe.) I don't think that works all that well, but "ranking" may 
> well refer to the usual "definite accept" to "definite reject" scale or "in 
> top 10% of papers". Given the tendency of the first ranking to concentrate 
> around the non-committal middle, the latter seems more helpful, but I'm not 
> sure that's a "best" practice.
>
> On May 30, 2013, at 11:40 AM, Ken Calvert <calv... at netlab.uky.edu> wrote:
>
>> Hi Joe -
>>
>> Good idea, thanks for doing this.  I think your proposal is pretty much on 
>> target.  Just a couple of thoughts on #6:
>>
>>> 6. paper review process            E/A/D
>>>
>>>     E = considers average rank AND outlier info, discussion points
>>>         also based on natural 'gap' in evaluation
>>>     A = considers average rank based on natural gap in evaluation
>>>     D = considers rank only
>>
>> (i) I interpret these criteria as referring to the accept/reject decision 
>> process, rather than the "paper review process".  Perhaps the title should 
>> be "acceptance decision process" or something like that?
>>
>> (ii) What about considering the transparency of the decision process?
>> I.e., whether all (or almost all) decisions are made with in full view of 
>> the TPC and with the TPC's approval or at least the opportunity to object.
>>
>> (iii) Can you please clarify what you mean by "natural gap in evaluation"? I 
>> would probably interpret this to mean that the accept/reject line is drawn, 
>> as far as possible, so that there is a clear gap between the (average 
>> ratings of) the accepted papers and the rejected papers.  But I don't think 
>> that's realistic - especially in large/general conferences, where there are 
>> papers from many areas, there will be not be a bright line in the 
>> ratings/rankings between rejected and accepted papers.  This also seems to 
>> conflict with "considers rank only" being Deficient.  So maybe I've just not 
>> understood what this means.
>>
>> Cheers,
>>
>> KC
>>
>> On 29 May 2013, at 14:05 PM, Joe Touch <to... at isi.edu> wrote:
>>
>>> Hi, all,
>>>
>>> As part of the ComSoc technical cosponsorship (TCS) process, TCs are
>>> supposed to nominate at least two members of the TPC who will monitor
>>> the review process.
>>>
>>> However, there doesn't appear to be any guidelines for providing
>>> feedback on that process.
>>>
>>> I've drafted the following, which I hope will open a discussion on
>>> this issue. If it evolves into something useful, perhaps it can be
>>> posted on the TC websites for use by those appointed to monitor
>>> TC-endorsed TCS'd meetings.
>>>
>>> NB: I've cross-posted this to TCCC, ITC, and TCHSN, which are where I
>>> participate primarily; if any other TC has suggestions, please take
>>> the discussion to the TCCC list if possible.
>>>
>>> Thanks,
>>>
>>> Joe
>>>
>>> -----------------------------------------------------------------
>>>
>>> Rating system:
>>>      EXCELLENT    best-practice to be aspired to
>>>      AVERAGE      acceptable practice
>>>      DEFICIENT    cause for concern for ComSoc involvement
>>>
>>> 1. TPC participation invitation        E/A/D
>>>
>>>     E = before first Call for Papers (CFP) issued
>>>     A = before CFP submissions due
>>>     D = after CFP submissions due
>>>
>>> 2. involvement in CFP promotion        E/A/D
>>>
>>>     E = invited to forward CFP and submit
>>>     A = invited to submit
>>>     D = neither
>>>
>>> 3. paper assignment for review        E/A/D
>>>
>>>     E = invited to select papers based on expertise and
>>>         abstracts/titles
>>>     A = invited to select based on topic area
>>>     D = not invited to select
>>>
>>>     NB: "everyone reviews all" = E
>>>
>>> 4. paper review format            E/A/D
>>>
>>>     E = includes rank, feedback for author, and private
>>>         feedback for TPC discussion
>>>     A = includes rank and author feedback
>>>     D = includes only rank
>>>
>>> 5. TPC meeting                E/A/D
>>>
>>>     E = in-person meeting with support for remote
>>>     A = in-person with no remote support or only telecon or e-mail
>>>     D = no meeting
>>>
>>> 6. paper review process            E/A/D
>>>
>>>     E = considers average rank AND outlier info, discussion points
>>>         also based on natural 'gap' in evaluation
>>>     A = considers average rank based on natural gap in evaluation
>>>     D = considers rank only
>>>
>>> 7. paper reviews returned        E/A/D
>>>
>>>     E = >=3 substantive reviews returned with rank and
>>>         comments for the authors
>>>     A = >=3 substantive reviews returned with rank and
>>>         at least a rationale for rejects
>>>     D = <3 reviews for some papers, reviews not returned at all,
>>>         or only rank provided
>>>
>>> 7. paper accept rate            E/A/D
>>>
>>>     E = <=50%, based on natural gap in paper evaluation
>>>     A = <=50%, not based on 'gap'
>>>     D = >50%
>>>
>>> ------------------------------------------------------
>>> _______________________________________________
>>> IEEE Communications Society Tech. Committee on Computer
>>> Communications
>>> (TCCC) - for discussions on computer networking and communication.
>>> Tccc at lists.cs.columbia.edu
>>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>
>> Ken Calvert
>> Professor and Chair, Computer Science Department Acting Director, Vis
>> Center University of Kentucky
>>
>>
>>
>>
>>
>> _______________________________________________
>> IEEE Communications Society Tech. Committee on Computer Communications
>> (TCCC) - for discussions on computer networking and communication.
>> Tccc at lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>
>
>
> _______________________________________________
> IEEE Communications Society Tech. Committee on Computer Communications
> (TCCC) - for discussions on computer networking and communication.
> Tccc at lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>
_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
 




More information about the Tccc mailing list