Skip to Content.
Sympa Menu

commons-research - Re: [Commons-research] open review model for upcoming workshop

commons-research AT lists.ibiblio.org

Subject: Commons-research mailing list

List archive

Chronological Thread  
  • From: Giorgos Cheliotis <gcheliotis.lists AT gmail.com>
  • To: commons-research AT lists.ibiblio.org
  • Subject: Re: [Commons-research] open review model for upcoming workshop
  • Date: Thu, 19 Feb 2009 14:51:57 +0800

Thanks for the thoughtful notes Philipp. You and Christopher have done enough to make me think more about the issue and as an editor, reviewer and a submitter of academic papers I assure you that I have seen my share of good and bad practices. I am even one to do research that is largely 'out-of-the-box' and have on occasion suffered the consequences. And I will be among the first to agree that there is something wrong with modern academic scholarship, but that's a longer discussion.

My gripe still is that in none of the arguments you or Horrobin presented did I find actual support for the hypothesis that an open and eponymous review process will generally produce better results than an anonymous peer review process. I find it interesting as a reviewing model, and do not want to outright dismiss it, but on the other hand saying that anonymous peer review may sometimes be flawed (or even that it may be deeply flawed), does not necessarily point to open and eponymous review as the solution.

Also, perhaps we should separate the two dimensions: one is open vs. closed and the other is eponymous vs. anonymous. A variety of combinations are possible... but I won't analyze this further here, I think you get my drift.

Let me just finish by saying that a research event is planned for this year and I hope to be able to make an announcement soon. It's happening in a different way from last year and the event itself will likely be different compared to what was originally envisioned. Sorry for not having been very 'open' about this yet, it's out of respect for the hosts of the event and their constraints. Last year we were able to secure support from CC and iCommons very fast and with no strings attached (for which I am thankful). That helped us make the whole planning process very open and inclusive. This year with no large event like the iSummit to latch on to it's a bit more complicated. Stay tuned :)


Giorgos

On Feb 19, 2009, at 12:34 AM, Philipp Schmidt wrote:

I haven't done a huge amount of research on this, but a quick scan
indicates that:

Anonymous peer-review might have been practiced for a long time, but
(1) it is far from being the only process in use by top journals (2)
and, at least in some cases, it does not work very well.

(1) For example, the British Medical Journal (bmj.com) has an open
peer review process that seems to serve them well.

(2) Horrobin (caveat: a well-known critic of the peer review process)
describes a study of neuroscience articles: "Their report should be
read in full; however, the conclusions are alarmingly clear. For one
journal, the relationships among the reviewers' opinions were no
better than that obtained by chance. For the other journal, the
relationship was only fractionally better. For the meeting abstracts,
the content of the abstract accounted for only about 10 to 20 percent
of the variance in opinion of referees, and other factors accounted
for 80 to 90 percent of the variance."

http://post.queensu.ca/~forsdyke/peerrev4.htm

I am undecided on the question of author's anonymity, since knowing
the author's identity has also been found to create a strong bias. In
one experiment (mentioned in "Irrationality") previously published
papers by scientists from top Universities, were re-submitted under
different names and rejected.

Nobody wants to do away with peer-review, but opening up the process
will increase accountability and quality of reviews (and hence of
accepted papers), rather than diminish them. And, since we are
studying "open" innovation, education, science it would be a good case
of practicing what we preach.

I rest my case here. Giorgios has done a great job pulling this event
together in Japan, and he is the driving force behind the US event.
The "open" world is a meritocracy, and since he has put in more sweat
equity than me, the decision is up to him. But I did feel it was
necessary to point out why an open peer-review approach might be
beneficial.

Best - P


On Thu, Feb 12, 2009 at 11:54 AM, Christopher Adams
<christopher.lee.adams AT gmail.com> wrote:
In the interests of keeping this venerable thread alive, I thought that I
would add a couple of thoughts that I had discussed off-list with Giorgios
awhile back.

In respects to the open comment policy of the last research workshop, I
myself am certainly guilty of neglecting this state of the process. I did
look through the accepted papers to see who would be attending (even going
so far as to contact two of the authors), but I did not leave any comments
online.

The failure to take advantage of this open process may have both human and
technical causes. But considering the community of scholars we hope to
interest in the next conference, might we not want to use this as an
opportunity to evolve how an open research workshop can be conducted?

I can envision a web-presence that would bring attention to the conference,
cultivate interest and frame it as an important academic event; further
still, provide a forum for prospective attendees to float paper ideas; offer
live-streaming and micro-blog participation for those unable to attend in
person; become a launchpad for further research; etc.

I suppose that we are all waiting for exact dates and venue for the
conference in August, but I'm hoping that others will step forward with new
proposals, soon enough .

- - kind regards
- - christopher adams

On Thu, Jan 8, 2009 at 5:08 PM, Alek Tarkowski <alek AT creativecommons.pl >
wrote:

Dear all,

The issue on how to setup the peer review process requires probably more
time and thought than I have right now... But two things came to my mind
reading the discussion:

Regarding Giorgos's last post, I'm not surprised no one commented
afterwards the abstracts. I don't think it's an issue of their length -
it's rather that they are not as meaningful as full texts, in my opinion
you only get a "feeling" of what the research / paper is about. Of
course, experiments with online commenting prove that people don't
comment too much at all (look for example at things done with the
Commentpress system - a plugin for Wordpress - by the Institute for the
Future of the Book).

Secondly, I am not sure about the argument that you need secrecy to have
healthy relationships in the community. It's plausible to be critical
without being offensive. I'm sure all of you do informal commenting /
reviews for fellow researchers / friends - and in those reviews I'm sure
you're sometimes critical. And it works. If the comments are sound,
what's there to foul a relationship?

I am not hundred percent sure about this - secrecy of reviews is such an
entrenched rule in our communities... The main difficulty I have with
open reviewing is that people might review "too nicely" - but on the
other hand blind reviewing allows for some level of un- accountability on
the part of the reviewer...

Having said all this, personally I do not find this *the* most important
issue - open reviewing is for me a nice addition to a portfolio of
"experiments with openness", but even with traditional reviewing things
work well (as proven by last years workshop). Though I think there is
always a level of randomness (and thus unfairness) to every selection
process.

Philipp, it would be good to know, how the open review process works for
the editors and authors of the journal you mentioned.

Best,

Alek


--
dr Alek Tarkowski
koordynator / public lead
Creative Commons Polska / Poland
http://creativecommons.pl

_______________________________________________
Commons-research mailing list
Commons-research AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/commons-research


_______________________________________________
Commons-research mailing list
Commons-research AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/commons-research





--
Jan Philipp Schmidt - "Sharing Nicely" at www.bokaap.net - "Hacking
Education" at www.peer2peeruniversity.org
_______________________________________________
Commons-research mailing list
Commons-research AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/commons-research





Archive powered by MHonArc 2.6.24.

Top of Page