FORCE2015 1K Challenge Results

Printer-friendly version

Add Comment

Results - See Discussion Below

Winner: Crowdreviewing: the Sharing Economy at its Finest
Submitter: Werner Liebregts, Utrecht University

The current review process of scientific articles is outdated. Only a few reviewers assess the quality of papers before publication. Many more experts read them after publication, have a strong opinion about their contents, but are hardly able to share it to a wide audience. Academics would greatly benefit from crowdreviewing, a post-publication peer review (PPPR) process in which anyone who has read a scientific article is asked to review it according to a standardized set of questions. Read More


Discussion: The Crowd & The 1K Challenge?

The three submissions that originally garnered the most voted did this by encouraging a large number of people to sign up for membership in FORCE11 and vote for their submission, in some cases friends and family who do not appear to have had a strong connection to the ideals of the organisation. This was a process that, while allowed by the rules, could be construed as contrary to the spirit of the competition, which is about engaging the FORCE11 community. In two cases, the authors of the proposals withdrew after this was pointed out to them; in one case the author decided to accept the money and participate in a discussion of the issues involved.

What we'd like to raise now with the community is the question of how to deal with this kind of issue. On the one hand, the $1k challenge programme has been extremely successful in the past in encouraging community-based ideas. On the other, the current arrangement is, arguably at least, open to abuse. How can we reconcile the desire to encourage crowd decision making (e.g. with a low barrier to entry) while ensuring the integrity of such competitions? Or is a completely open vote perfectly viable?

 
What's your opinion? Please comment below. 
 
Note: We would still like to award the two additional 1k challenge prizes. Based on the discussion below, we'll decide how to proceed. 
 

Comments

I'm looking forward to seeing what our community thinks on what happened.

Do not open voting to everyone and limit it to Force11 members.

Of all of my thoughts, I'm most impressed by how the 1K challenge committee is handling this issue so transparently and with integrity. It's so refreshing! Kudos! As far as what to do, I think it's important to stay within the spirit of the competition. Ideally, all authors would be people of integrity, so this would not be a problem. I applaud those that withdrew after the fact was pointed out to them. I applaud even more those that didn't solicit outside the community in the first place. As for how to proceed from here, I don't know. Pick the next highest two, hoping that the same abuse didn't occur, or... revote?

That's a good question. The ideas were all really good. Maybe a revote for the next two as you suggest.

Suggesting that I didn't act with integrity... I have had many positive reactions from people all around the world, who probably voted for my entry (possibly after signing up with FORCE11, but not because I asked them to do that). I even got several offers to put my idea into action on existing platforms. All this strengthens me in my belief that I submitted a good idea (which is, so far, underexposed).

Moreover, a vast majority of the very few people that I solicited to vote for my entry are colleagues, thus being part of academia, and they are all in favor of bringing my idea into practice. And, isn't the challenge also a way to grow the FORCE11 community?

No one knows who would have won the competition if voting would have been restricted to those who were already member of FORCE11 before voting started. It could have been me (as well), it could have been someone else.

By the way, I am really curious about how many votes came from already "active" members before the voting procedure started. Not a lot it seems.

Would still be interested to know what "very few" means. More than a year later, when going through the membership directory I find 41 people from the Netherlands who have signed up between 17 and 27 February 2015, including 8 family members and 11 members of the Utrecht School of Economics where you work. But my guess would be that these 41 were all sollicited and there may have been more who since ended their Force11 membership and of course those that did not act on the plea to vote for this idea. No one can prove how many of these voted for your idea. My guess would be 41. Deduct that from 46 votes. Still, no rules were broken and I believe that entering the challenge was done with full integrity. It's just that solliciting should not play a role in deciding what idea holds the most for Force11 and its goals. Finally, I am still interested to learn how you would put the ideas into practice.

May be you can restrict the voting rights only to the members who joined FORCE11 before the competition started.

Good idea! This would have been my preference.

Maybe a % of the score could be allowed for new members with logins aka family and friends and votes vetted as only one person per outlest. The
measure of how social media is engaged is part of making the process on new innovation  work but peer view is also important so this could offer
the best of both worlds. The 2 next highest plus the withdrawn sector could participate in a run off?

I think this is an interesting idea to kind of mix the social media metrics with the votes of members. I guess there's a lot we could do "technically" to prevent this but it also makes us less open...

I think one of the issues facing all of open scholarship is whether we encourage good behavior through rules or through norms.  A natural response to what happened in the 1K challenge might be to put in place tougher rules and checks designed to thwart attempts to game the system.  But my feeling is that communities like FORCE11 should be able to develop norms that are enforced by people monitoring their own behavior.  So we should be able to make the principles of community explicit and people should have to agree to abide by those principles.  But it is an experiment in "open".  

Ironically, the winning entry was about an open peer review system where, one might argue, it could be subject to the same type of solicited positive reviews.   

 

Could be indeed. Unfortunately, entries were restricted to a maximum number of words, so I was not able to elaborate on how I would like to deal with this disadvantage of an open peer review process.

Could you elaborate on that now? I think maybe one is the issue of breadth of a community? Who has expertise to comment? What do you think?

Part of the idea is that reviewing should be made accessible for all who have read an article (i.e. for a crowd, it is in the name). These are probably people with a certain level of expertise in the field of research where the article belongs to. The fact that it is a crowd rules out the possibility of relatives and/or close friends having a major influence on the overall assessment of the article's quality. Linking this to the challenge: if all of your 1,243 active members would have been active in voting, then no one would have been able to "game the system" by canvassing voters.

That's an interesting assumption about whether the number of peers will outweigh non peers. I thinks it's unclear if this will be the case. 

In addition, you make the process fully open, such that everyone knows who has reviewed what (hence, not like the challenge, which has been called 'open' erroneously). You can even create a system in which reviewing is honored and/or rewarded in some way. In the end, you voluntarily contribute to academia by doing that.

Maryann, I had the immediate same thought when I read about this: "Ironically, the winning entry was about an open peer review system". So, I think that this is a more general problem than the one about voting for the 1K challenge.  As part of Force11 mission, one of the discussions that we should have is the validity of open review (perhaps in the form of a workshop or a group). This could include a review of the existing data on what works and what doesn't, and the mechanisms one could implement to make a transparent, open process valid.

For now, I think nobody should get the 1K until this is resolved - it should be considered a fraud voting, if we know that's the case.

I guess the issue is how to define where the edges of a crowd are, i.e. who's in and who's out. I imagine myself in this scenario i.e. encouraging friends and family to vote on something like this. Clearly you could discount my mum (although I guess not the case who have followed their parents into a similar line of work!) Some of my friends are broadly in the same professional space so their votes might count, but they still might be voting because I'd asked them to rather than because they had considered each proposal in detail and chosen mine objectively. 

I'm curious as to whether it's immediately evident somehow from the results set that something like this has happened, in which case it's probably OK to carry on with the current process just indicating that you will discount any votes that are clearly 'inappropriate'? Or, if it's hard to spot, and was only spotted by chance in this case, a more preventive approach might be required, such as Amy's idea above.

It feels like this situation must have occurred somewhere else </beetles off to dig around the web for similar examples>

PS agree that the openness of your approach is great - thanks!

This is my understanding... So we had setup the system so that only FORCE members could vote. But we couldnt trace which people voted for particular projects. We could see in the logs the pickup in the number of people and who they are. We could try to put something more traceable in place but at the time we didn't think it would be an issue.

This is a very difficult issue. On the one hand, the 1k model is a great way of promoting ideas--the idea of small microgrants on a minimal application process is great and the idea that the winners would some how reflect the interests of the society is also extremely good. But on the other, an open voting system to select the winners is fundamentally open to being gamed. It is quite literally a popularity contest and so is almost inescapably going to end up with the rigour of a home coming king and queen competition.

Maybe what we need to do is examine what the core features of the competition are and then look to see how we could preserve their essential elements. In my view, the core attributes of the 1k challenge are the following:

  1. It is a light competition that supports blue sky thinking (so very minimal application process and very free budget rules)
  2. It is responsive to developing community interests (i.e. it is not regulated by an inner circle, but by peers)
  3. It is strongly associated with the Force11 conference and so good at capturing developing ideas.

Given this, I think any solution that moves us away from these three strengths is less than optimal. And in fact, I'd argue that the current voting system actually moves us away from the third attribute (i.e. it separates the results from conference) and, because it can be gamed, also probably moves us away from the second (i.e. the results may not actually represent developing community interests, but rather popularity and influence networks within the community. If in order to overcome this problem, we introduce a ton of rules, then we are going to be violating the first core attribute, the lightness of the competition.

My suggestion would be that in future we deal with the 1k challenge actually at the conference. I.e. we present and vote on the winners at the closing session in much the same way we do the other prizes. If we announce the competition long in advance, working groups and others can prepare ideas before the conference meets, and then these ideas can be fleshed out on site by participants as the conference meets, with a final presentation of the ideas before the final session. Voting could then be by hand or using some kind of clicker system (there are systems that allow you to use phones and other devices). If we used clickers, then people following the stream or twitter feed could also vote.

This wouldn't stop the competition being a popularity conference, but it would tie it much more closely to the conference itself (value 3) and the popularity contest would be among people actually following the conference (value 2). Because people would need to digest the proposals, the applications would have to be very light and readable (value 1). It would be also a lot harder to game, because voting would require people to be active in the organisation at a specific moment and it is probably harder to get your grandfather and aunt to vote for you if they have to sit through the closing session of the conference in order to vote.

As for the picking the remaining winners. I'm in a conflict as I have a proposal in the mix. But I'd say we might want to do a revote using the mailing list of the day the contest was announced.

It might not be clear from what I said that I think an important part of running the competition at the conference would be that it would also spark new ideas from the actual sessions themselves. In both conferences I've attended, I've met new people and had ideas for new collaborations onsite. If the 1k competition was understood as a funding mechanism for great ideas coming out of the conference, then it would encourage people to develop these as well.

You could see it in that regard as being almost like a hackathon for funding.

+1 I really like the that it comes from the conference and is decide at there. 

At the last Beyond the PDF, the ideas had to come from the conference;  indeed, that was the intent of the 1K challenge.  Collaborations and ideas that come up at the conference needed some impetus to continue beyond the conference and the 1K Challenge was a way to do that.  We did not accept submissions after the conference, although the voting was done after the conference.  But perhaps doing the entire challenge at the conference, including remote attendees I'm assuming, would be the way to go.

I like the idea of deciding at the conference. We do need to support remote participation so I still might envision that happening by voting through the website.

I agree you'd somehow need to support the virtual attendees and somehow in a way that wasn't then just as open to the gaming. 

The FORCE Exec set up the Challenge and specified that voting would be the way to decide winners, but left unstated what the exact rules were.  

Several people used this vagueness to canvass additional votes for their projects and ended up winning.  These additional voters have no intrinsic connection to the goals of FORCE11.  However, they all became were legitimately registered FORCE members at the time of voting, and they each voted themselves (it looks like).  So none of the official rules were violated.  

In principle, openness implies the freedom to canvass whoever you want.  

But many people feel that something went wrong, because the people responding to the canvassing are not people who (we suspect) are interested in the community’s goals and are not likely to continue playing a role in it.  So their joining and voting was a one-time occurrence, performed simply to enable the winners to win.  

So what?, says one point of view.  They joined to vote, and voted, and that was their act of belonging to the community.  

So no, says another.  Many of the conventions that structure our societies are just conventions, not rules, invented and evolved to make the community functional.  No-one says you have to thank people for services or assistance, or make way for others on the sidewalk.  It is at the heart of volunteer/enthusiast organizations like FORCE11 to explore new conventions/ways of working together in order to improve society, in our case more-open scholarly practice.  This involves respecting one another’s specializations, accomplishments, ideas, suggestions, etc., even if they don’t really directly affect our everyday lives.  The 1K Challenge was designed to elicit suggestions for some such mechanisms/inventions/ideas.  The prize money was a nice afterthought, not the point.  

The act of canvassing people to join just in order to win the Challenge perverts the basis of voting.  We as a community are no longer sure that the winning ideas are those deemed best among the volunteers/enthusiasts of FORCE11.  So the efforts of the Exec to set up the Challenge, obtain the prize money, etc., are all wasted.  

This is the root of my personal discomfort: The FORCE Exec set up the Challenge in a genuine desire to elicit information about good ideas, good according to the people interested in the goals of FORCE11. The canvassing of voters from people not centrally interested in FORCE11 thwarted this desire.  The canvassers who grasped this consequence of their actions and withdrew their participation from the competition thereby affirmed their adherence to the desire behind the Challenge. The canvassers who did not withdraw despite being given the chance have been given a chance to explain the reasoning behind their decision.  There are several options:
- perhaps it was to make a point about the inherent vulnerability of open voting systems without specific rules of participation
- perhaps it was the glory of winning
- perhaps it was the money

Linking this to publication:

It is the stated opinion of one of the winners that open review/judgment of publication, especially post-publication, is a better system than we have today (closed review by selected experts).  It seems to me that this might be the case as long as these post-publicaiton judges are in fact experts.  But if they have been canvassed to generate reviews by the author from his/her family and friends, that undercuts the whole idea of informed review.  I do not want to live in a world where my scientific papers are judged by the family members of someone in my field simply because he or she asks them to.  And I doubt that the winner, or any of us, wants to either. 

 

I would like to add another option: the sincere belief that I deserved to win.

Your entire comment is based on the speculative thought that many people voted for my entry because of canvassing, and that these people will not contribute to the community's goals ever again. You and I simply do not know. Neither how many people that I personally know voted for me (I can think of only a few) nor whether or not they will continue contributing to the community. I sincerely hope they will do that, but that is up to them.

Outside of what may or may not have happened in this particular instance (i.e., the 1K Challenge),  I would appreciate your comments on the larger issue of open vs closed that Ed raised above.  Both systems are subject to various types of injustice.  The question is whether one system is demonstrably better than the other and how would we know?  For the system you propose, what are the outcome measures that would be used to determine whether the crowdreviewing is giving us better than we have now?  I think that the entire community has this onus when proposing something new:  how do we know it's better?  

See one of my comments above.

To me this sums it all up. This is not about rules, but about ethics. I never understood canvassing. I get a lot of requests to vote for something. They all go "please vote for our project to win this competition". I never act on that, even if they come from my best fiends or colleagues. I simply do not grap what they are thinking when they are asking around. I also feel offended and not taken seriously, as in my book they approach me because they think I may be willing to act unethically. When I ask them about that they mostly tell me not to take it so seriously, that it's just a game. 

So let's make this less of a game. My suggestion would be to - even if we keep this fully within the conference or restricted to pre-contest members - go a little bit beyond voting and ask voters to add three lines of what they think are the TWO best entries and why they think one the these two really is the best. That way it will be more difficult to canvass and the community gets an idea of what people see in a project and how it might fit the goals of Force11.

I am one of the FORCE11 members that voted for the crowdreviewing proposal of Werner LiebregtsI have to admit I am a bit surpised about the tone of this discussion. I very much appreciate any open discussion on how to improve the voting system for the 1K challenge, but I would refrain from talking about "abuse" in this context. The voting was in full compliance with the current terms and conditions. Please correct me if I'm wrong.

It's the organizer's task to phrase clear and unequivocal terms and to create the technical requirements to restrict the pool of persons entitled to vote to the desired size. Surely, we can argue whether it was intended or desired that people register at FORCE11 just to vote for a specific project. It was clearly not forbidden. And if we eventually decide that mommy and daddy shouldn't be allowed to vote for their children's proposal, then the terms and conditions should mention that explicitly and both shouldn't be able to register at FORCE11 in a twinkled.

No system is perfect, but gladly, we can improve it over time. I think there are already many good suggestions above, e.g. the idea to restrict voting to members that joined before the challenge started.

What I am trying to say is that we shouldn't focus on punishing the legitimate winners of the current voting system (which we implemented) but focus on how to change the rules for the next challenge. I think there is no moral obligation to withdraw any proposal.

You're definitely right we should learn from this. Also in some sense this is about not having clear rules, which comes from the fact  that FORCE11 tends to be pretty light weight. 

I don't think the winners were punished. We just pointed out the issue that we had and why we felt .. um.. uncomfortable.

I hope that Werner can really make a good project with the 1k. Maybe this discussion can help him?

But it's also important talking about this I the open - otherwise we're not the open community I think we all want.

Tone  is a tricky one - it's  hard online. Personally I follow W3C and Wikipedia maxim of the assumption of good faith. So yes let's focus on the issues and not on the person and how we can improve.

 

Thanks for this, Sebastian! I totally agree.

Somehow I completely missed the above discussion last year. I come back to it now because while looking at the member directory I saw so many Force members from my own university that I never come across in local discussions on scholarly communication. I used the member directory to find out if a kind of Dutch satellite event for the Scholalrly Commons working group was a viable idea. That's how I found that exactly half of the 82 Dutch members registered between 17 and 27 February last year, during the 1K Challenge voting. I also saw that apart from 8 family members, 11 voters were recruited from the Utrecht School of Economics where Werner works. Without repeating all that has already been said and led to a new system in 2016 I would like to list how this (still) compromises the functioning of Force11.

  1. The directory is polluted with people who probably are not very interested in Force11, let alone become active for it.
  2. The directory has fake names
  3. The directory has people that registered twice
  4. The directory has many people without affiliation
  5. People may have thought that extra outreach for Force11 in the Netherlands was not necessary because of the high number of members, thus preventing real progress for Force11 here
  6. Planning an event is more difficult if the directory is polluted

One final thought I would like to share: in the aftermath of Force2015 I was quite surprised to see so many votes for Werners idea, which I thought was not that special, also because it lacked any ideas on how to reach its goal. I recall having seconds thoughts at the time about how smart Force11 people were if they flocked to support this idea. But with perhaps 41 of 46 votes being recruited I now am glad to say that after all Force people are a smart bunch, so apologies for having second thoughts back then. This is how the unethical voting behaviour compromised and still compromises our community.

Hi Jeroen,

Thanks for pointing this out and the ramifications of this to the proper functioning of FORCE11. I think we may need to do something about our membership directory.

thanks,

Paul