Skip to Content.
Sympa Menu

commons-research - Re: [Commons-research] open review model for upcoming workshop

commons-research AT lists.ibiblio.org

Subject: Commons-research mailing list

List archive

Chronological Thread  
  • From: Giorgos Cheliotis <gcheliotis.lists AT gmail.com>
  • To: commons-research AT lists.ibiblio.org
  • Subject: Re: [Commons-research] open review model for upcoming workshop
  • Date: Thu, 8 Jan 2009 16:53:40 +0800

Thanks Christopher for sharing your thoughts. We actually did what you mention for the last workshop, not during the review process, but after accepted papers (abstracts back then) were posted online. No one wrote a comment on a posted abstract as far as I know. In principle I see nothing against repeating this, even if I expect that again there will be virtually no comments. Perhaps if we have full papers online there will be more discussion. I think one difference may be that all of us are used to commenting on short, easily digestible, content online. Reading a scholarly paper, even an extended abstract, and commenting on it requires a more serious commitment. But then those with that level of commitment should anyway attend the workshop, so I guess they tend to reserve their arguments for the workshop itself. Maybe in addition to the papers themselves we can have say a blog of some form where anyone can post an opinion piece, shorter, and easier to grasp, maybe more provocative, relating to the topics of the workshop, but not so academic - such pieces may incite more comments... and could generate useful discussions in parallel to the more academic part of the event. Anyway, these are just thoughts... it's easy to come up with ideas, it's difficult to make them really work. 

As for those asking whether there are news about the event itself, the only news is that the venue is still not confirmed. I said Stanford in an earlier email. That may not be the final venue after all. We hope to resolve this soon, I understand that it has taken a while. 

On Jan 8, 2009, at 10:32 AM, Christopher Adams wrote:

Giorgios,


Personally, from the perspective an author of a paper at the Research Workshop in Sapporo, I found the review process well-handled. I do not have such intimate knowledge of the review process, but I find the reasoning that you have laid out rather persuasive.


Nevertheless, I do empathize with the desire to make peer review ever more open. One technically feasible solution would be to activate an open comment system for accepted papers, wherein the reviewers would be encouraged, and the public invited, to critique the papers in a public forum.


In this way the reviewers would not have to guard their comments to the author, while still allowing a means by which substantive debate could be made public.


| christopher adams


On Wed, Jan 7, 2009 at 6:47 PM, Giorgos Cheliotis <gcheliotis.lists AT gmail.com> wrote:
Thanks Philipp for the contribution to what has been a very quiet
mailing list lately (but this will change as we should soon shift
gears with planning for the next workshop/conference).

Personally I don't find much benefit in such a move, but it is
something to consider, if there is enough support for getting rid of
the anonymity in reviews and enough strong arguments to back up such a
decision.

My thoughts are in the exactly opposite direction: if anything, I
think this year we could try to go with double-blind reviews, which is
the standard for many of the more serious events and journals in
several academic fields, and for good reason. There are many such good
reasons why anonymity is valuable in such situations where one is
asked to judge the fruits of another's often long and painstaking
labor, i.e. when we judge works that people invest a lot of time,
effort, money and ego in. I think I do not need to list out all the
possible scenarios where lack of anonymity can go horribly wrong and
compromise the integrity of reviews, or, destroy relationships which
are badly needed for any academic community to sustain itself.

As for accountability, let's remember that the reviewers are never
entirely anonymous. They are known to the chairs and to the PC
members. I believe a certain degree of accountability is preserved
under double-blind reviews, as PC members and chairs can control for
each other's integrity as well as for the quality of submitted
reviews. Both myself and Tyng-Ruey for example prompted many reviewers
last year to improve the quality of their reviews, or to reconsider
cases where we felt their reviews were not entirely fair. There was
significant juggling involved in achieving anonymity as many PC
members were also authors of submitted papers, but I think we did a
good job, especially Tyng-Ruey who was in charge of coordinating
reviews (as he had not submitted a paper himself, so had no direct
conflict of interest).

Now if an author feels that all of a PC is colluding or is otherwise
corrupt, then that author may just as well avoid submitting to that
venue and/or write a letter of complaint, or take whatever other
action with respect to any specific claims that he/she wants to make.
In special cases I'm sure a process can be followed to investigate any
claims of unfair treatment of a paper. But I for one want to be able
to call a poor paper "poor" and still be friends with the author. If
my assessment is not sufficiently supported by arguments and I just
call it "poor" for no good reason, the opinions of the other reviewers
and PC members should weigh in and I should be called upon by them to
provide justification for my assessment. But making the reviews public
will not make me more accountable, it will make me more timid and
perhaps in some cases less honest.

Ok, well, these are my thoughts, sorry for the long post... let's keep
talking. In short, publishing reviews for accepted papers can still be
embarassing to all, as a paper can be accepted even with a very
critical review. Anonymity aims not only at limiting the potential for
public embarassment, it is more about safeguarding the relationships
that we build with each other, as well as our ability to honestly
critique each other's work. I have yet to see convincing arguments to
the contrary.


On Jan 7, 2009, at 3:52 PM, Philipp Schmidt wrote:

> I am participating in an open peer review process for the Human
> Resources for Health journal. I think something like this would be a
> good model for the upcoming commons research workshop/meeting as it
> would address many of the concerns about being too open that were
> raised prior to the iSummit 2008. I am forwarding the instructions for
> review, but have removed information that allows identification of the
> article.
>
> The review state is closed to the public (I suppose to avoid
> embarrassment of researchers whose papers might be rejected) but open
> insofar as authors receive reviews of their paper along with the names
> of the reviewers. Once a paper has been accepted all reviews (with
> names of reviewers) and the paper are published online.
>
> Best - P
>
>
> ---------- Forwarded message ----------
> From: HRH Editorial <hrhjournal AT who.int>
> Date: Wed, Jan 7, 2009 at 9:43 AM
> Subject: Review of Human Resources for Health manuscript
> To: Philipp Schmidt <phi.schmidt AT gmail.com>
>
>
> Human Resources for Health
> Methodology
>
> Dear Dr Schmidt,
>
> Many thanks for agreeing to review the above manuscript, which has
> been submitted to Human Resources for Health. Please let us know
> whether there is likely to be any difficulty in meeting the deadline
> for your report, which is 28 January 2009.
>
> Below you will find instructions for retrieving the manuscript and
> returning your report via our website. Please remember that the
> information in this manuscript is confidential.
>
> As we mentioned in our previous message, we operate an open peer
> review process for this journal in order to give peer reviewers credit
> for their work and to ensure accountability. This means that we ask
> reviewers to agree to their signed report being passed on to the
> authors. If the manuscript is accepted for publication, your signed
> report will be posted on the website along with the article and the
> other reviewers' reports. We ask you to confirm that you consent to
> this at the end of the peer review report.
>
> Many thanks for all your help with this manuscript.
>
> With best wishes,
>
>
> The HRH Editorial Team
>
> Tel: +44 (0)20 7323 0323
> Facsimile: +44 (0)20 7631 9923
> e-mail: hrhjournal AT who.int
> Web: http://www.human-resources-health.com/
> _______________________________________________
> Commons-research mailing list
> Commons-research AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/commons-research

_______________________________________________
Commons-research mailing list
Commons-research AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/commons-research

_______________________________________________
Commons-research mailing list
Commons-research AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/commons-research




Archive powered by MHonArc 2.6.24.

Top of Page