Skip to Content.
Sympa Menu

commons-research - Re: [Commons-research] open review model for upcoming workshop

commons-research AT lists.ibiblio.org

Subject: Commons-research mailing list

List archive

Chronological Thread  
  • From: Philipp Schmidt <phi.schmidt AT gmail.com>
  • To: commons-research AT lists.ibiblio.org
  • Subject: Re: [Commons-research] open review model for upcoming workshop
  • Date: Wed, 18 Feb 2009 18:34:17 +0200

I haven't done a huge amount of research on this, but a quick scan
indicates that:

Anonymous peer-review might have been practiced for a long time, but
(1) it is far from being the only process in use by top journals (2)
and, at least in some cases, it does not work very well.

(1) For example, the British Medical Journal (bmj.com) has an open
peer review process that seems to serve them well.

(2) Horrobin (caveat: a well-known critic of the peer review process)
describes a study of neuroscience articles: "Their report should be
read in full; however, the conclusions are alarmingly clear. For one
journal, the relationships among the reviewers' opinions were no
better than that obtained by chance. For the other journal, the
relationship was only fractionally better. For the meeting abstracts,
the content of the abstract accounted for only about 10 to 20 percent
of the variance in opinion of referees, and other factors accounted
for 80 to 90 percent of the variance."

http://post.queensu.ca/~forsdyke/peerrev4.htm

I am undecided on the question of author's anonymity, since knowing
the author's identity has also been found to create a strong bias. In
one experiment (mentioned in "Irrationality") previously published
papers by scientists from top Universities, were re-submitted under
different names and rejected.

Nobody wants to do away with peer-review, but opening up the process
will increase accountability and quality of reviews (and hence of
accepted papers), rather than diminish them. And, since we are
studying "open" innovation, education, science it would be a good case
of practicing what we preach.

I rest my case here. Giorgios has done a great job pulling this event
together in Japan, and he is the driving force behind the US event.
The "open" world is a meritocracy, and since he has put in more sweat
equity than me, the decision is up to him. But I did feel it was
necessary to point out why an open peer-review approach might be
beneficial.

Best - P


On Thu, Feb 12, 2009 at 11:54 AM, Christopher Adams
<christopher.lee.adams AT gmail.com> wrote:
> In the interests of keeping this venerable thread alive, I thought that I
> would add a couple of thoughts that I had discussed off-list with Giorgios
> awhile back.
>
> In respects to the open comment policy of the last research workshop, I
> myself am certainly guilty of neglecting this state of the process. I did
> look through the accepted papers to see who would be attending (even going
> so far as to contact two of the authors), but I did not leave any comments
> online.
>
> The failure to take advantage of this open process may have both human and
> technical causes. But considering the community of scholars we hope to
> interest in the next conference, might we not want to use this as an
> opportunity to evolve how an open research workshop can be conducted?
>
> I can envision a web-presence that would bring attention to the conference,
> cultivate interest and frame it as an important academic event; further
> still, provide a forum for prospective attendees to float paper ideas; offer
> live-streaming and micro-blog participation for those unable to attend in
> person; become a launchpad for further research; etc.
>
> I suppose that we are all waiting for exact dates and venue for the
> conference in August, but I'm hoping that others will step forward with new
> proposals, soon enough .
>
> - - kind regards
> - - christopher adams
>
> On Thu, Jan 8, 2009 at 5:08 PM, Alek Tarkowski <alek AT creativecommons.pl>
> wrote:
>>
>> Dear all,
>>
>> The issue on how to setup the peer review process requires probably more
>> time and thought than I have right now... But two things came to my mind
>> reading the discussion:
>>
>> Regarding Giorgos's last post, I'm not surprised no one commented
>> afterwards the abstracts. I don't think it's an issue of their length -
>> it's rather that they are not as meaningful as full texts, in my opinion
>> you only get a "feeling" of what the research / paper is about. Of
>> course, experiments with online commenting prove that people don't
>> comment too much at all (look for example at things done with the
>> Commentpress system - a plugin for Wordpress - by the Institute for the
>> Future of the Book).
>>
>> Secondly, I am not sure about the argument that you need secrecy to have
>> healthy relationships in the community. It's plausible to be critical
>> without being offensive. I'm sure all of you do informal commenting /
>> reviews for fellow researchers / friends - and in those reviews I'm sure
>> you're sometimes critical. And it works. If the comments are sound,
>> what's there to foul a relationship?
>>
>> I am not hundred percent sure about this - secrecy of reviews is such an
>> entrenched rule in our communities... The main difficulty I have with
>> open reviewing is that people might review "too nicely" - but on the
>> other hand blind reviewing allows for some level of un-accountability on
>> the part of the reviewer...
>>
>> Having said all this, personally I do not find this *the* most important
>> issue - open reviewing is for me a nice addition to a portfolio of
>> "experiments with openness", but even with traditional reviewing things
>> work well (as proven by last years workshop). Though I think there is
>> always a level of randomness (and thus unfairness) to every selection
>> process.
>>
>> Philipp, it would be good to know, how the open review process works for
>> the editors and authors of the journal you mentioned.
>>
>> Best,
>>
>> Alek
>>
>>
>> --
>> dr Alek Tarkowski
>> koordynator / public lead
>> Creative Commons Polska / Poland
>> http://creativecommons.pl
>>
>> _______________________________________________
>> Commons-research mailing list
>> Commons-research AT lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/commons-research
>
>
> _______________________________________________
> Commons-research mailing list
> Commons-research AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/commons-research
>
>



--
Jan Philipp Schmidt - "Sharing Nicely" at www.bokaap.net - "Hacking
Education" at www.peer2peeruniversity.org




Archive powered by MHonArc 2.6.24.

Top of Page