[Tccc] ComSoc technical cosponsorship - r...
Joe Touch
touchatisi.edu
Thu May 30 16:20:08 EDT 2013
On 5/30/2013 11:32 AM, Henning Schulzrinne wrote:
> Also, ranking could be seen as meaning that a TPC member ranks papers
> within their review portfolio (paper #7 is best, #8 second best").
> (Infocom tried this, I believe.) I don't think that works all that
> well, but "ranking" may well refer to the usual "definite accept" to
> "definite reject" scale or "in top 10% of papers". Given the tendency
> of the first ranking to concentrate around the non-committal middle,
> the latter seems more helpful, but I'm not sure that's a "best"
> practice.
Agreed; I will correct that to refer to "absolute" rankings.
Joe
>
> On May 30, 2013, at 11:40 AM, Ken Calvert <calv... at netlab.uky.edu> wrote:
>
>> Hi Joe -
>>
>> Good idea, thanks for doing this. I think your proposal is pretty much on
>> target. Just a couple of thoughts on #6:
>>
>>> 6. paper review process E/A/D
>>>
>>> E = considers average rank AND outlier info, discussion points
>>> also based on natural 'gap' in evaluation
>>> A = considers average rank based on natural gap in evaluation
>>> D = considers rank only
>>
>> (i) I interpret these criteria as referring to the accept/reject decision
>> process, rather than the "paper review process". Perhaps the title should
>> be "acceptance decision process" or something like that?
>>
>> (ii) What about considering the transparency of the decision process?
>> I.e., whether all (or almost all) decisions are made with in full view of
>> the TPC and with the TPC's approval or at least the opportunity to object.
>>
>> (iii) Can you please clarify what you mean by "natural gap in evaluation"? I
>> would probably interpret this to mean that the accept/reject line is drawn,
>> as far as possible, so that there is a clear gap between the (average
>> ratings of) the accepted papers and the rejected papers. But I don't think
>> that's realistic - especially in large/general conferences, where there are
>> papers from many areas, there will be not be a bright line in the
>> ratings/rankings between rejected and accepted papers. This also seems to
>> conflict with "considers rank only" being Deficient. So maybe I've just not
>> understood what this means.
>>
>> Cheers,
>>
>> KC
>>
>> On 29 May 2013, at 14:05 PM, Joe Touch <to... at isi.edu> wrote:
>>
>>> Hi, all,
>>>
>>> As part of the ComSoc technical cosponsorship (TCS) process, TCs are
>>> supposed to nominate at least two members of the TPC who will monitor
>>> the review process.
>>>
>>> However, there doesn't appear to be any guidelines for providing
>>> feedback on that process.
>>>
>>> I've drafted the following, which I hope will open a discussion on this
>>> issue. If it evolves into something useful, perhaps it can be posted on
>>> the TC websites for use by those appointed to monitor TC-endorsed TCS'd
>>> meetings.
>>>
>>> NB: I've cross-posted this to TCCC, ITC, and TCHSN, which are where I
>>> participate primarily; if any other TC has suggestions, please take the
>>> discussion to the TCCC list if possible.
>>>
>>> Thanks,
>>>
>>> Joe
>>>
>>> -----------------------------------------------------------------
>>>
>>> Rating system:
>>> EXCELLENT best-practice to be aspired to
>>> AVERAGE acceptable practice
>>> DEFICIENT cause for concern for ComSoc involvement
>>>
>>> 1. TPC participation invitation E/A/D
>>>
>>> E = before first Call for Papers (CFP) issued
>>> A = before CFP submissions due
>>> D = after CFP submissions due
>>>
>>> 2. involvement in CFP promotion E/A/D
>>>
>>> E = invited to forward CFP and submit
>>> A = invited to submit
>>> D = neither
>>>
>>> 3. paper assignment for review E/A/D
>>>
>>> E = invited to select papers based on expertise and
>>> abstracts/titles
>>> A = invited to select based on topic area
>>> D = not invited to select
>>>
>>> NB: "everyone reviews all" = E
>>>
>>> 4. paper review format E/A/D
>>>
>>> E = includes rank, feedback for author, and private
>>> feedback for TPC discussion
>>> A = includes rank and author feedback
>>> D = includes only rank
>>>
>>> 5. TPC meeting E/A/D
>>>
>>> E = in-person meeting with support for remote
>>> A = in-person with no remote support or only telecon or e-mail
>>> D = no meeting
>>>
>>> 6. paper review process E/A/D
>>>
>>> E = considers average rank AND outlier info, discussion points
>>> also based on natural 'gap' in evaluation
>>> A = considers average rank based on natural gap in evaluation
>>> D = considers rank only
>>>
>>> 7. paper reviews returned E/A/D
>>>
>>> E = >=3 substantive reviews returned with rank and
>>> comments for the authors
>>> A = >=3 substantive reviews returned with rank and
>>> at least a rationale for rejects
>>> D = <3 reviews for some papers, reviews not returned at all,
>>> or only rank provided
>>>
>>> 7. paper accept rate E/A/D
>>>
>>> E = <=50%, based on natural gap in paper evaluation
>>> A = <=50%, not based on 'gap'
>>> D = >50%
>>>
>>> ------------------------------------------------------
>>> _______________________________________________
>>> IEEE Communications Society Tech. Committee on Computer Communications
>>> (TCCC) - for discussions on computer networking and communication.
>>> Tccc at lists.cs.columbia.edu
>>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>
>> Ken Calvert
>> Professor and Chair, Computer Science Department
>> Acting Director, Vis Center
>> University of Kentucky
>>
>>
>>
>>
>>
>> _______________________________________________
>> IEEE Communications Society Tech. Committee on Computer Communications
>> (TCCC) - for discussions on computer networking and communication.
>> Tccc at lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>
>
_______________________________________________
IEEE Communications Society Tech. Committee on Computer Communications
(TCCC) - for discussions on computer networking and communication.
Tccc at lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc
More information about the TCCC
mailing list