Regarding Berkman/Sciences Po study

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Regarding Berkman/Sciences Po study

Dario Taraborelli-3
I’d like to give everybody on this list some information on the Berkman/Sciences Po research project that many of you have been discussing here.

On Thursday the Wikimedia Foundation announced the launch of a banner to support a study led by a team at the Berkman Center/Sciences Po and recruiting participants from the English Wikipedia editor community [1]. The banner was taken down within hours of its launch after concerns raised in various community forums (the Admin Noticeboard [2], the Village Pump Tech [3], various IRC channels and mailing lists such as foundation-l [4] and internal-l [5]) that the design was confusing, that it was perceived as a commercial ad and that the community approval process and privacy terms were unclear and hardly visible.

Here’s what happened until the launch, what went wrong after the launch and what we are planning to do next.

==The prequel==
This proposal went through a long review process, involving community forums, the Research Committee and various WMF departments since early 2010.

The Berkman research team first approached WMF to discuss this study in January 2010. They suggested a protocol to recruit English Wikipedia contributors to participate in an early version of this study by March 2010 and posted a proposal to the Administrators’ noticeboard to get community feedback [6]. The community response at that time opposed the proposed recruitment protocol (posting individual invitation messages on user talk pages). It was suggested instead that the recruitment should be handled through a CentralNotice banner to be displayed to registered editors, but concerns were raised on how to minimize the disruption.

To address these concerns, the proposal went through a full review with the Wikimedia Research Committee, that was completed in July 2011. The RCom evaluated the methods, the recruitment strategy, the language used in the survey and approved the proposal pending a final solution for the recruitment taking into account the concerns expressed by the community [7].

Based on suggestions made by community members (e.g. [8]) the research team started to work on a technical solution to selectively display a banner to a subset of registered editors of the English Wikipedia meeting certain eligibility conditions. WMF agreed to invest engineering effort into a system that would allow CentralNotice to serve contents to a specific set of editors –  functionality that would benefit future campaigns run by the community, chapters or the Foundation [9] [10].

A new CentralNotice backend was then designed to look up various editor metrics (i.e. number of contributions, account registration date and editor privileges) – all public information available from our database – and to perform a participant eligibility check against these metrics. A banner would then be displayed to eligible participants, posting the above data (user ID + editor metrics) along with a unique token to the server hosting the survey upon clicking. On the landing page of the survey, participants would have the possibility to read the privacy terms of the survey and decide whether to take it or not.

Throughout the review process of this recruitment protocol, the research team received constant feedback from the Foundation’s legal team, the community department, the tech department and the communication team before the campaign went live.

The campaign was announced in the CentralNotice calendar one month before its launch [11] and the launch was with a post on the Foundation’s blog. The banner was enabled on December 8 at 11:00pm UTC. 800+ participants completed the study within a few hours since its launch. The banner was then taken down by a meta-admin a few hours after the launch due to the concerns described above.  

So what went wrong?

==A few explanations we owe you==

• Is the Foundation running ads?
No, this banner is a recruitment campaign for a research project that has been thoroughly reviewed by the Research Committee. We have a long tradition of supporting recruitment for research about our communities via various sitenotices. The methodology of this project is sound and the recruitment method less invasive than thousands of individual messages posted on user talk pages. We believe this research will help advance our understanding of the dynamics of participation in our projects. Receiving support by the Research Committee implies that all published output and anonymized data produced by this study will be made available under open licenses. [12] The banner also received full Wikimedia Foundation approval before its launch.

• Is this campaign conflicting with the fundraiser?
No, this banner is running only for a subset of logged-in editors for whom the main fundraiser campaign has already been taken down. We carefully timed this campaign to minimize the impact on the fundraiser and we scheduled it on the CentralNotice calendar with a month notice for this reason.

• Is this campaign running at 100% on the English Wikipedia?
No, the banner has been designed to target a subsample of the English Wikipedia registered editor population. Based on estimates by the research team, the eligibility criteria apply to about 10,000 very active contributors and about 30,000 new editors of the English Wikipedia. The target number of completed responses is 1500.

• Why does the banner include logos of organizations not affiliated with Wikimedia?
The design of the banner was based on the decision to give participants as much information as possible about the research team running the project and to set accurate expectations about the study.  


==What we are doing now==

We realize that despite an extensive review, the launch of this project was not fully advertised on community forums. We plan to shortly resume the campaign (for the time needed by the researchers to complete their responses) after a full redesign of the recruitment protocol in order to address the concerns raised by many of you over the last 24 hours. Here’s what we are doing:

• Provide you with better information about the project
We asked the research team to promptly set up a FAQ section on the project page on Meta [13], and to be available to address any concern about the study on the discussion page of this project. The project page on Meta will be linked  from the recruitment banner itself.

• Redesign the banner
We understand that the banner design has been interpreted by some as ad-like (even if the goal was to make clear that this study was not being run by WMF, as it implied a redirection to a third party website for performing the experiment). In coordination with the research team, we will come up with a banner design that will be more in line with the concerns expressed by the community (for instance by removing the logos from the banner).

• Make privacy terms as transparent as possible
Upon clicking on the banner, participants accept to share their username, edit count and user privileges with the research team. The previous version didn’t make it explicit and we are working to address this problem. To make the process totally transparent we will make the acceptance of these terms explicit in the banner itself.

Once redirected to the landing page, participants will have to accept the terms of participation in order to enter the study. The project is funded by the European Research Council: the data collected in this study is subject to strict European privacy protocols. The research team will use this data for research purposes only. The research team is not exposed to and does not record participants’ IP addresses.
 
==How you can help==

We would like to hear from you on the redesign of the banner to make sure it meets the expectations of the community and doesn’t lend itself to any kind of confusion. We will post the new banners to Meta and try to address all pending questions before we resume the campaign.
 
This is one of the first times we’re supporting a complex, important research initiative like this one, and I apologize for the bumps in the road. We believe that supporting research is part of our mission: it helps advance our understanding of ourselves. So thanks again for all support you can give in making this a success.


Dario Taraborelli
Senior Research Analyst, Wikimedia Foundation
 
[1] http://blog.wikimedia.org/2011/12/08/experiment-decision-making/ 
[2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incidents#Harvard.2FScience_Po_Adverts
[3] http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Search_banner_Wikipedia_Research_Committee
[4] http://lists.wikimedia.org/pipermail/foundation-l/2011-December/070742.html
[5] https://lists.wikimedia.org/mailman/private/internal-l/2011-December/018842.html
[6] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archive222#Researchers_requesting_administrators.E2.80.99_advices_to_launch_a_study
[7] http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interactions_and_Behavior#RCom_review
[8] http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065580.html
[9] http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065558.html
[10] http://meta.wikimedia.org/wiki/CentralNotice_banner_guidelines
[11] http://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&oldid=3056067
[12] http://meta.wikimedia.org/wiki/Research:Subject_recruitment
[13] meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and_Behavior


_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Berkman/Sciences Po study

geni
On 10 December 2011 03:49, Dario Taraborelli <[hidden email]> wrote:
> We plan to shortly resume the campaign

What do I have to do to stop you?

> • Provide you with better information about the project

I'm not remotely interested. There have been rather a lot of studies
done over the years. They rarely lead to anything useful and by far
the most interesting ones didn't ask any questions at all. The
community (which practically is a lot smaller than you seem to think)
has long since started to suffer from  study fatigue.


> • Redesign the banner
> • Make privacy terms as transparent as possible

Missing the point.


--
geni

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Berkman/Sciences Po study

Thomas Morton
In reply to this post by Dario Taraborelli-3
Hi Dario,

This proposal went through a long review process, involving community

> forums, the Research Committee and various WMF departments since early 2010.
>
> The Berkman research team first approached WMF to discuss this study in
> January 2010. They suggested a protocol to recruit English Wikipedia
> contributors to participate in an early version of this study by March 2010
> and posted a proposal to the Administrators’ noticeboard to get community
> feedback [6]. The community response at that time opposed the proposed
> recruitment protocol (posting individual invitation messages on user talk
> pages). It was suggested instead that the recruitment should be handled
> through a CentralNotice banner to be displayed to registered editors, but
> concerns were raised on how to minimize the disruption.
>

This is not a good summary of the conclusions there at all; and it is
worrying that it has been read that way...

You seem to have taken that discussion as implicit approval to run a
CentralNotice banner - although that was certainly suggested as an option
at the time I think it was reasonably expected for further community input
later down the road. Certainly when I supported the suggestion of some sort
of targeted site notice I envisioned a text link, or something.


> Throughout the review process of this recruitment protocol, the research
> team received constant feedback from the Foundation’s legal team, the
> community department, the tech department and the communication team before
> the campaign went live.
>

But not the community?


> The campaign was announced in the CentralNotice calendar one month before
> its launch [11] and the launch was with a post on the Foundation’s blog.
> The banner was enabled on December 8 at 11:00pm UTC. 800+ participants
> completed the study within a few hours since its launch. The banner was
> then taken down by a meta-admin a few hours after the launch due to the
> concerns described above.
>

Again; not announced to the community. There was a clear an present
communication failure here.


> We realize that despite an extensive review, the launch of this project
> was not fully advertised on community forums. We plan to shortly resume the
> campaign (for the time needed by the researchers to complete their
> responses) after a full redesign of the recruitment protocol in order to
> address the concerns raised by many of you over the last 24 hours. Here’s
> what we are doing:
>
> • Provide you with better information about the project
> We asked the research team to promptly set up a FAQ section on the project
> page on Meta [13], and to be available to address any concern about the
> study on the discussion page of this project. The project page on Meta will
> be linked  from the recruitment banner itself.
>
> • Redesign the banner
> We understand that the banner design has been interpreted by some as
> ad-like (even if the goal was to make clear that this study was not being
> run by WMF, as it implied a redirection to a third party website for
> performing the experiment). In coordination with the research team, we will
> come up with a banner design that will be more in line with the concerns
> expressed by the community (for instance by removing the logos from the
> banner).
>
> • Make privacy terms as transparent as possible
> Upon clicking on the banner, participants accept to share their username,
> edit count and user privileges with the research team. The previous version
> didn’t make it explicit and we are working to address this problem. To make
> the process totally transparent we will make the acceptance of these terms
> explicit in the banner itself.
>
> Once redirected to the landing page, participants will have to accept the
> terms of participation in order to enter the study. The project is funded
> by the European Research Council: the data collected in this study is
> subject to strict European privacy protocols. The research team will use
> this data for research purposes only. The research team is not exposed to
> and does not record participants’ IP addresses.
>

You need to tell this *to the community*. Otherwise the discussion will
simply strike up again once you re-enable it. I notice you posted this
exact same message to wikipedia-en-l. The lack of recent discussion on that
list should tell you how effective that is as a communication tool.

The vast majority of English Wikipedia discussion occurs on-wiki, and the
vast majority of editors prefer discussions to occur on-wiki. If you want
to interact with the community, and in this case I think you have to, then
you really have to do so on-wiki :)

We would like to hear from you on the redesign of the banner to make sure
> it meets the expectations of the community and doesn’t lend itself to any
> kind of confusion. We will post the new banners to Meta and try to address
> all pending questions before we resume the campaign.
>

Most en.wiki editors don't hang out on Meta - and I think it is reasonable
not to expect them to. Especially as this is purely a project focused on
English Wikipedia; it needs to be discussed on English Wikipedia. If you
need help with the right places/protocol then I would be happy to oblige.

This once again highlights the huge disconnect between Wikipedians, the
foundations and the various higher level commitees. It's the same every
time - something big appears, the community get cross/upset/confused, the
foundation etc. express incredulity... and no one talks.

My final comment is this; I am fairly active around here. I am on
foundation-l, I read meta for important notices, and I am active on
en.wiki. And the first time I knew about this (since the Admin Noticeboard
discussion some months back) was when the banner appeared. So how anyone
else stood a chance of giving input I do not know :) Whilst much of the
discussion was public, it was "hidden". And the key failing here is in not
making it open and accessible.

Tom
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Berkman/Sciences Po study

Jonathan Cardy
In reply to this post by geni
@Geni. http://meta.wikimedia.org/wiki/Research:Omnibus_Survey was my
preferred alternative, but it was considered unacceptable by the Research
Committee. Are you really determined to stop such research altogether or
could you compromise on one annual survey?

Cheers

WereSpielChequers


On 10 December 2011 10:31, geni <[hidden email]> wrote:

> On 10 December 2011 03:49, Dario Taraborelli <[hidden email]>
> wrote:
> > We plan to shortly resume the campaign
>
> What do I have to do to stop you?
>
> > • Provide you with better information about the project
>
> I'm not remotely interested. There have been rather a lot of studies
> done over the years. They rarely lead to anything useful and by far
> the most interesting ones didn't ask any questions at all. The
> community (which practically is a lot smaller than you seem to think)
> has long since started to suffer from  study fatigue.
>
>
> > • Redesign the banner
> > • Make privacy terms as transparent as possible
>
> Missing the point.
>
>
> --
> geni
>
> _______________________________________________
> WikiEN-l mailing list
> [hidden email]
> To unsubscribe from this mailing list, visit:
> https://lists.wikimedia.org/mailman/listinfo/wikien-l
>
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Berkman/Sciences Po study

geni
On 10 December 2011 10:53, WereSpielChequers
<[hidden email]> wrote:
> @Geni. http://meta.wikimedia.org/wiki/Research:Omnibus_Survey was my
> preferred alternative, but it was considered unacceptable by the Research
> Committee. Are you really determined to stop such research altogether or
> could you compromise on one annual survey?
>
> Cheers
>
> WereSpielChequers


I can't realistically stop an annual survey. However if you did have
an annual survey I'm pretty sure this wouldn't have been bundled into
it. rather too meta.

--
geni

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Regarding Berkman/Sciences Po study

James Farrar
In reply to this post by Dario Taraborelli-3
On 10 December 2011 11:38, James Farrar <[hidden email]> wrote:

> Here's the thing.
>
> Banner adverts are bad.
>
> Sometimes they're necessary (the fundraiser being the most obvious
> example, but other "get involved with Wikipedia/WMF/chapters" stuff
> qualifies) - but when they're not, they shouldn't be tolerated.
>
> Recruiting for a third party's research project is advertising.
> [[WP:PROMOTION]] (point 1) is very clear on the matter: it is not
> appropriate.
> On Dec 10, 2011 3:50 AM, "Dario Taraborelli" <[hidden email]>
> wrote:
>


> [deleted because apparently my little text added at the top when I posted
> from my phone pushed the message over the list's pathetically small limit.
> That was 24 hours ago; apologies for the delay, I couldn't get to a
> computer until now.]
>
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l