
GROUP HAS COMPLETED THEIR WORK AND IS NO LONGER ACTIVE.
FOR QUERIES, PLEASE WRITE TO ‘INFO@FORCE11.ORG’
Crowdreviewing: The Sharing Economy at its Finest
Submitted by: Werner Liebregts
The current review process of scientific articles is outdated. Only a few reviewers assess the quality of papers before publication. Many more experts read them after publication, have a strong opinion about their contents, but are hardly able to share it to a wide audience. Academics would greatly benefit from crowdreviewing, a post-publication peer review (PPPR) process in which anyone who has read a scientific article is asked to review it according to a standardized set of questions. Today’s consumers themselves decide whether the consumed good meets their quality standards, and this is quickly and easily shared with the rest of the world. Others can then let their demand for the reviewed good depend on the reviews. Interested people observe the articles’ quality (and the reasons why) at a glance. Visibility of the reviews, including comments and remarks, will even have an upward effect on the quality of future research. The sharing economy at its finest, applied to academia.
Project Goal
In short: to successfully run a pilot project on crowdreviewing scientific articles.
ScienceOpen has offered me the opportunity to run a pilot project on their already existing platform for open access (OA) publishing, for which I want to thank them a lot.
A major goal of ScienceOpen is to foster the OA movement by raising the awareness for high-quality OA and by involving the entire scientific community in a transparent and open evaluation process. Today, the ScienceOpen platform comprises nearly 1.5 million OA articles and preprints so far, aggregated from various sources.
The associated ScienceOpen network enables authors, readers and reviewers to connect and keep informed about recently published OA articles in their field(s) of interest. ScienceOpen explicitly aims to provide services to the whole scientific community. That is:
- To authors, by enabling them to publish their results immediately, and to receive transparent feedback;
- To readers, by offering them a free to use database of OA articles;
- To reviewers, by giving them the opportunity to receive recognition and credit for their valuable and voluntary duty.
More specifically, I am going to set up a collection of scientific OA articles in economics. A collection should start with a minimum of ten articles. Applying a complete yet accessible review process to it should result in a high number of people reviewing each article in a consistent way. If successful, a new metric for the quality of an individual scientific article might be derived.
Project Outcomes
The outcome of the project is threefold:
- Agreement on how to review scientific articles in a good way;
- A standardized scientific review process suitable for crowdreviewing;
- A first proof of concept.
The first project outcome will be obtained by facilitating and moderating online discussions concerning the scientific review process. The conclusions drawn from the discussions should then be incorporated into the new standard way of reviewing scientific articles. Applying this process to a collection at ScienceOpen will (hopefully) lead to the third project outcome. The success of this project also depends on your cooperation and input!
Project Timeline
Below, you can find a rough timeline of the project, starting with the moment of idea generation in November last year, and ending with a presentation at the next FORCE11 conference.
- November 2014: Idea generation
- November 2014 – end: Idea promotion (incl. personal and Skype meetings with partners)
- February 2015: The £1k Challenge
- February 2015 – end: Contact with ScienceOpen
- March 2015: Editor status at ScienceOpen
- May 2015: Collection set up at ScienceOpen
- May 2015 – August 2015: Online discussions about the scientific review process
- September 2015 – end: Idea implementation
- April 2016: Presentation of the project outcomes (/intermediate results) at the FORCE2016 meeting in Portland
The process of idea implementation comprises the application of an agreed-upon review process to the collection of articles set up at ScienceOpen, inviting readers to review at least one of those articles, monitoring the progress (and making adjustments if necessary), and evaluating the entire process. Short updates about the progress will be published on a regular basis.
Discussions
You are cordially invited to actively participate in online discussions about how to review scientific articles in a good way. Several discussion forums will be opened one by one in the upcoming months. Any comment and/or suggestion is highly appreciated. Examples of questions that will be addressed are:
- What hinders and what encourages people to do a review?
- Can we come up with a standardized set of questions (or, assessment criteria) that are applicable to any type of scientific article from any field of research or should we have multiple ways of reviewing?
- Is it possible to create a one-dimensional indicator of an article’s quality based on reviews?
- What is the minimum number of people to be able to call it a ‘crowd’? In other words, when can the results from crowdreviewing be called reliable?
- Should the reviewing process be fully open? Why (not)?
- Should reviewing be rewarded? If so, how?
Comments
Posted by Stephanie Hagstrom | May 4, 2015
Adding Comment from deWaard Blog regarding 1k Challenge
Are the Crowds Really All That Wise? What the FORCE11 1k Challenge Taught us About Crowdreview
Posted by Eduard Hovy | May 6, 2015
A response to Liebregts
Mr. Liebregts, one of the winners of the 1k challenge, proposes PPPR as a model to 'fix' the peer reviewing process. There are several flaws with this model and with the eay it is proposed.
We have all felt the sting of disappointment when our (oh so excellent) submission was rejected by incompetent or just shoddy reviewers. And sometimes we’ve also felt also the compensatory (although somehow never quite adequate) satisfaction when the same submission is accepted elsewhere.
How to fix the reviewing process?
One can take at least four distinct approaches:
Is PPPR a valid option? While it sounds appealingly democratic, there are at least three problems that make this unworkable and rather naive:
This discussion is about policy, and is political, not scientific. Mr. Liebregts obtained his $1K challenge prize by a process of canvassing, and is exercising his right to argue his point. He is providing us as academics a chance to exercise clear thinking and responsible reflection of one of our time-honed practices.
Posted by Werner Liebregts | May 8, 2015
Coming up: discussion forums
Dear Eduard, I'm about to set up the first discussion forum about part of the issues that you've raised. Will you be involved in the discussions to also answer the questions that you posed?
Posted by Eduard Hovy | May 8, 2015
1K PPPR Discussion
It depends; if the discussion is thoughtful and not self-promoting or stupid, then I would be happy to contribute. But if it is simply an exercise in sloppy wishful thinking then I will not participate.
So far I see no valid answers to the following problems with PPPR:
- why peers would take time to create new reviews post-publication (real peers, not the author's own graduate students and friends and family)
- who would select these peers and check over their reviews to remove problematic content
- where the reviews would be hosted, for how long, and how they would be organized
- who would pay attention to these reviews post-publication, since traditional citation counts are already giving much of the information we want
Posted by Werner Liebregts | May 11, 2015
Answers
Let's try to find these answers together, Eduard! I already have my own thoughts (of course), but I do not want to harm the discussion by giving my own opinion in advance. You are right in saying that it should not be a self-promoting discussion, but a fully objective and open one instead. At the same time, this also means that I expect none of the discussants to be narrow-minded or prejudiced.