[Commons-research] open review model for upcoming workshop

Christopher Adams christopher.lee.adams at gmail.com
Wed Jan 7 21:32:29 EST 2009


Giorgios,


Personally, from the perspective an author of a paper at the Research
Workshop in Sapporo, I found the review process well-handled. I do not have
such intimate knowledge of the review process, but I find the reasoning that
you have laid out rather persuasive.


Nevertheless, I do empathize with the desire to make peer review ever more
open. One technically feasible solution would be to activate an open comment
system for *accepted* papers, wherein the reviewers would be encouraged, and
the public invited, to critique the papers in a public forum.


In this way the reviewers would not have to guard their comments to the
author, while still allowing a means by which substantive debate could be
made public.


| christopher adams


On Wed, Jan 7, 2009 at 6:47 PM, Giorgos Cheliotis <
gcheliotis.lists at gmail.com> wrote:

> Thanks Philipp for the contribution to what has been a very quiet
> mailing list lately (but this will change as we should soon shift
> gears with planning for the next workshop/conference).
>
> Personally I don't find much benefit in such a move, but it is
> something to consider, if there is enough support for getting rid of
> the anonymity in reviews and enough strong arguments to back up such a
> decision.
>
> My thoughts are in the exactly opposite direction: if anything, I
> think this year we could try to go with double-blind reviews, which is
> the standard for many of the more serious events and journals in
> several academic fields, and for good reason. There are many such good
> reasons why anonymity is valuable in such situations where one is
> asked to judge the fruits of another's often long and painstaking
> labor, i.e. when we judge works that people invest a lot of time,
> effort, money and ego in. I think I do not need to list out all the
> possible scenarios where lack of anonymity can go horribly wrong and
> compromise the integrity of reviews, or, destroy relationships which
> are badly needed for any academic community to sustain itself.
>
> As for accountability, let's remember that the reviewers are never
> entirely anonymous. They are known to the chairs and to the PC
> members. I believe a certain degree of accountability is preserved
> under double-blind reviews, as PC members and chairs can control for
> each other's integrity as well as for the quality of submitted
> reviews. Both myself and Tyng-Ruey for example prompted many reviewers
> last year to improve the quality of their reviews, or to reconsider
> cases where we felt their reviews were not entirely fair. There was
> significant juggling involved in achieving anonymity as many PC
> members were also authors of submitted papers, but I think we did a
> good job, especially Tyng-Ruey who was in charge of coordinating
> reviews (as he had not submitted a paper himself, so had no direct
> conflict of interest).
>
> Now if an author feels that all of a PC is colluding or is otherwise
> corrupt, then that author may just as well avoid submitting to that
> venue and/or write a letter of complaint, or take whatever other
> action with respect to any specific claims that he/she wants to make.
> In special cases I'm sure a process can be followed to investigate any
> claims of unfair treatment of a paper. But I for one want to be able
> to call a poor paper "poor" and still be friends with the author. If
> my assessment is not sufficiently supported by arguments and I just
> call it "poor" for no good reason, the opinions of the other reviewers
> and PC members should weigh in and I should be called upon by them to
> provide justification for my assessment. But making the reviews public
> will not make me more accountable, it will make me more timid and
> perhaps in some cases less honest.
>
> Ok, well, these are my thoughts, sorry for the long post... let's keep
> talking. In short, publishing reviews for accepted papers can still be
> embarassing to all, as a paper can be accepted even with a very
> critical review. Anonymity aims not only at limiting the potential for
> public embarassment, it is more about safeguarding the relationships
> that we build with each other, as well as our ability to honestly
> critique each other's work. I have yet to see convincing arguments to
> the contrary.
>
>
> On Jan 7, 2009, at 3:52 PM, Philipp Schmidt wrote:
>
> > I am participating in an open peer review process for the Human
> > Resources for Health journal. I think something like this would be a
> > good model for the upcoming commons research workshop/meeting as it
> > would address many of the concerns about being too open that were
> > raised prior to the iSummit 2008. I am forwarding the instructions for
> > review, but have removed information that allows identification of the
> > article.
> >
> > The review state is closed to the public (I suppose to avoid
> > embarrassment of researchers whose papers might be rejected) but open
> > insofar as authors receive reviews of their paper along with the names
> > of the reviewers. Once a paper has been accepted all reviews (with
> > names of reviewers) and the paper are published online.
> >
> > Best - P
> >
> >
> > ---------- Forwarded message ----------
> > From: HRH Editorial <hrhjournal at who.int>
> > Date: Wed, Jan 7, 2009 at 9:43 AM
> > Subject: Review of Human Resources for Health manuscript
> > To: Philipp Schmidt <phi.schmidt at gmail.com>
> >
> >
> > Human Resources for Health
> > Methodology
> >
> > Dear Dr Schmidt,
> >
> > Many thanks for agreeing to review the above manuscript, which has
> > been submitted to Human Resources for Health. Please let us know
> > whether there is likely to be any difficulty in meeting the deadline
> > for your report, which is 28 January 2009.
> >
> > Below you will find instructions for retrieving the manuscript and
> > returning your report via our website. Please remember that the
> > information in this manuscript is confidential.
> >
> > As we mentioned in our previous message, we operate an open peer
> > review process for this journal in order to give peer reviewers credit
> > for their work and to ensure accountability. This means that we ask
> > reviewers to agree to their signed report being passed on to the
> > authors. If the manuscript is accepted for publication, your signed
> > report will be posted on the website along with the article and the
> > other reviewers' reports. We ask you to confirm that you consent to
> > this at the end of the peer review report.
> >
> > Many thanks for all your help with this manuscript.
> >
> > With best wishes,
> >
> >
> > The HRH Editorial Team
> >
> > Tel: +44 (0)20 7323 0323
> > Facsimile: +44 (0)20 7631 9923
> > e-mail: hrhjournal at who.int
> > Web: http://www.human-resources-health.com/
> > _______________________________________________
> > Commons-research mailing list
> > Commons-research at lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/commons-research
>
> _______________________________________________
> Commons-research mailing list
> Commons-research at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/commons-research
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ibiblio.org/pipermail/commons-research/attachments/20090108/fc4396c5/attachment.html 


More information about the Commons-research mailing list