discussion about wikipedia surveys

classic Classic list List threaded Threaded
55 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Heather Ford-3
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Aaron Halfaker-2
Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Dario Taraborelli-3
Hi all,

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

Is RCom an official WMF body or a group of volunteers?

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

Is RCom a body regulating subject recruitment?

No, subject recruitment was only one among many areas of interest identified by its participants [2]

Is RCom still alive?

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

What about requests for private data?

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

How can we incentivize researchers to “give back” to the community?

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

Dario


On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Lane Rasberry
Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
I am still thinking about what should be done with research.

yours,



On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:
Hi all,

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

Is RCom an official WMF body or a group of volunteers?

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

Is RCom a body regulating subject recruitment?

No, subject recruitment was only one among many areas of interest identified by its participants [2]

Is RCom still alive?

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

What about requests for private data?

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

How can we incentivize researchers to “give back” to the community?

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

Dario


On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--
Lane Rasberry
user:bluerasberry on Wikipedia
206.801.0814
[hidden email]

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Piotr Konieczny-2
In reply to this post by Heather Ford-3
This time I'll respond below.

On 7/29/2014 17:50, Heather Ford wrote:
+1 on Piotr's comments. 

And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months. 

I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments: 

1. developing ethical research guidelines for Wikipedia research
- by building on the WP:Ethically_researching_Wikipedia page and regularly pointing people to it

Two ideas:
* there's a drive to print out leaflets for Wikimania, this page could be advertised there
* even better, we should try to advertise it in a leaflet form at Wikisym
* WMF could try to create a short handout booklet based on it


2. finding ways of making responsible requests to the WMF for data that they hold that might benefit research outside the WMF
- through an official process with guidelines from the WMF on response times/ viable requests etc.


It is a good example of an idea that helps rather than hinders researchers, and an area where RCOM-like body assistance would be useful.

3. developing opportunities for researchers to collaborate and share what they're doing with the wider research community
- reorganising the research hub and pointing to best case practices etc (similar to the WP Global Education program, as Piotr suggests)
- actively recruiting WP researchers to join this list and visit the research hub
- some other regular way of involving researchers such as inviting them to showcase their work and have it recognised on the list, on the hub etc 
- recognising outstanding research (through a prize perhaps as Aaron suggested)


All +1

--
Piotr Konieczny, PhD
http://hanyang.academia.edu/PiotrKonieczny
http://scholar.google.com/citations?user=gdV8_AEAAAAJ
http://en.wikipedia.org/wiki/User:Piotrus


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Piotr Konieczny-2
In reply to this post by Aaron Halfaker-2
I have replied at https://en.wikipedia.org/wiki/Wikipedia_talk:Research_recruitment#Strong_objection

While this group is transparent, Wikipedia discussion is even more so, and I prefer to held discussion in a more transparent venue where possible.

For those that don't want to read another site's discussion, a short summary of my points:
* I agree that it's good to VOLUNTARILY recommend best practices as Aaron lists in his 1-3 points
* I don't recognize RCOM's or Aaron's authority to say things like "maybe you shouldn't be allowed to contact Wikipedians". If a researchers violates Wikipedia rules, our regular policies enforced by regular admin corp, plus in extreme cases potential shaming of unethical research through publicity/contacting unethical researcher departments should be enough.

--
Piotr Konieczny, PhD
http://hanyang.academia.edu/PiotrKonieczny
http://scholar.google.com/citations?user=gdV8_AEAAAAJ
http://en.wikipedia.org/wiki/User:Piotrus
On 7/29/2014 22:26, Aaron Halfaker wrote:
I don't think that it is appropriate that those who benefit from deregulation (e.g.  No oversight for running surveys.  No formalized community review process.) make the decisions about what is worth regulating.  You'll notice that the proposed policy that Poitr calls "instruction creep" basically states that you do three things:

1. Document your research.  Specifically, your methods of recruitment, consent process, data storage and publication strategy.
2. Discuss your research -- with Wikipedians to make sure that you won't cause a disruption
3. Proceed as consensus emerges.  

We all seem to agree that this is good practice.  Where is the rest of the "instruction creep"?  Where is the anti-researcher bend?  

Poitr, you speculate about potential problems like people just coming to say "IDONTLIKEIT", but I have yet to see that happen in RCOM's process despite the fact that we invite editors from the population being sampled to the conversation.  Even if it was true, I think that if some of your potential participants don't like what you are doing, you ought to address their concerns.  

I'm all for developing guidelines (note that Ethically researching Wikipedia IS NOT a guideline).  I've wrote my fair share of essays to help researchers & Wikipedians find their way around research projects in Wikipedia.  E.g. http://en.wikipedia.org/wiki/Wikipedia:Research and and https://en.wikipedia.org/wiki/User:EpochFail/Don%27t_bite_the_researchers.  However, I've watched good research projects fail because researchers didn't have the wikipedian backgrounds that you guys do (Heather and Poitr).  See some examples of (IRB approved) studies running into project-halting difficulties: https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Research#Examples_of_unmediated_interactions  These examples are what got me to start working on developing a process in the first place.

If you really think that documenting your research and having a discussion about it is too much instruction, then maybe you shouldn't be allowed to contact Wikipedians.  If you do think that every research project that does recruitment should be documented and discussed, why not just say so?

-Aaron


On Tue, Jul 29, 2014 at 3:50 AM, Heather Ford <[hidden email]> wrote:
+1 on Piotr's comments. 

And very, very happy to hear about https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia -- I think this is definitely the way to go: developing guidelines that we *regularly point people to* when they have questions etc. And maybe something that we as a group can work on in the coming months. 

I'll reiterate my suggestions for goals here and add some of Piotr's and others' comments: 

1. developing ethical research guidelines for Wikipedia research
- by building on the WP:Ethically_researching_Wikipedia page and regularly pointing people to it

2. finding ways of making responsible requests to the WMF for data that they hold that might benefit research outside the WMF
- through an official process with guidelines from the WMF on response times/ viable requests etc.

3. developing opportunities for researchers to collaborate and share what they're doing with the wider research community
- reorganising the research hub and pointing to best case practices etc (similar to the WP Global Education program, as Piotr suggests)
- actively recruiting WP researchers to join this list and visit the research hub
- some other regular way of involving researchers such as inviting them to showcase their work and have it recognised on the list, on the hub etc 
- recognising outstanding research (through a prize perhaps as Aaron suggested)

Looking forward to hearing Phoebe's suggestions!

Best,
Heather.

On 29 July 2014 09:04, Pine W <[hidden email]> wrote:

The good and bad news is that the status quo with RCOM is likely to remain unless someone in WMF, the Board, or the community is interested enough in addressing the situation to put in some effort to make RCOM a functioning organization.

At the moment I have the impression that WMF researchers are absorbing most of the work that RCOM and some dedicated RCOM admin support could do, like help with lit review and prevent outside researchers from using WMF databases in ways that compromise user privacy. My perception is that the current situation is inefficient for WMF and for outside researchers who want to do good work with WMF  or community resources, and also that RCOM lacks the resources to respond in timely ways to requests for help with outside research that could benefit Wikimedia. So, I there are reasons to changs the status quo, and I hope WMF or the Board would be interested in something like the proposal I made previously.

Phoebe, what do you think?

Pine


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Piotr Konieczny-2
In reply to this post by Aaron Halfaker-2
This an interesting clarification. I support framing RCOM's mission as educational (teaching researchers about best practices), and even more so clearly stating that its procedures are voluntary. In other words, such a body should have an uncontroversial consultative/advisory role, rather then be a gatekeeper of sorts. That said, I don't know if we need a "body" at all. Why couldn't all of this be done under existing community auspices such as WikiProject Research?

I still think our priority should be to redesign our research pages, create a proper research portal with best practices (and hopefully some carrot-like tools that help researchers, from certificates to how-tos for grants/data to research tools) that we could then advertise among most Wikipedia researchers.

IMHO one of RCOM's biggest fallacies was (is...) trying to frame itself as a gatekeeper then a facilitator.

--
Piotr Konieczny, PhD
http://hanyang.academia.edu/PiotrKonieczny
http://scholar.google.com/citations?user=gdV8_AEAAAAJ
http://en.wikipedia.org/wiki/User:Piotrus
On 7/29/2014 22:42, Aaron Halfaker wrote:
I don't believe there is any claim of authority for RCOM.  At least I was not involved in making claims that it is required and I do not see it as such.  In fact, I have argued in the past that studies run by Wikipedians won't gain much from the process[1]. However, I do recommend that academics -- especially those who do not otherwise engage with Wikipedians -- to work with an RCOM member to coordinate a review in order to ensure that you won't see massive push-back when you start recruiting on Wikipedia -- as studies tended to see when they were run before the process.  


On Tue, Jul 29, 2014 at 8:35 AM, Nathan <[hidden email]> wrote:
Hi Aaron, what's the source of authority for RCOM (or its members acting independently) to perform a review procedure and claim it is required? 



On Tue, Jul 29, 2014 at 9:27 AM, Aaron Halfaker <[hidden email]> wrote:
Re. RCOM and review processes, these are two different things.   RCOM is an old, defunct WMF sanctioned working group of staff, researchers and Wikipedians.  If we want to revive RCOM, it seems like this should be discussed in another thread.  


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Piotr Konieczny-2
In reply to this post by Dario Taraborelli-3
That's extremely helpful, and I suggest copying it to the https://meta.wikimedia.org/wiki/Research:Committee page

(that page needs many updates)
--
Piotr Konieczny, PhD
http://hanyang.academia.edu/PiotrKonieczny
http://scholar.google.com/citations?user=gdV8_AEAAAAJ
http://en.wikipedia.org/wiki/User:Piotrus
On 7/30/2014 07:00, Dario Taraborelli wrote:
Hi all,

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

Is RCom an official WMF body or a group of volunteers?

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

Is RCom a body regulating subject recruitment?

No, subject recruitment was only one among many areas of interest identified by its participants [2]

Is RCom still alive?

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

What about requests for private data?

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

How can we incentivize researchers to “give back” to the community?

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

Dario


On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Piotr Konieczny-2
In reply to this post by Lane Rasberry
Lane, how many survey requests do you get per year? And how much time do you spend on them?

Because myself, being in the Top 100 most active editors and thus I'd think fitting in your group of " about 30 super busy people", I get about ~2 requests per year and they cost me few minutes at most, which even being "super busy" I find I can afford.

What I'd focus with a call for the researchers (perhaps another idea for best practices) would be to ask people to do proper lit review. I don't think we have too many surveys, but I do think we have a not-too-small percentage of them pointlessly replicating prior research (as in - we probably don't need a n-th paper on Wikipedian's motivations that badly...). Of course, people who can't be bothered to to a proper lit review can't probably be bothered to find out about our best practices guides, even if we clean the mess that our research pages are currently, so... :/

--
Piotr Konieczny, PhD
http://hanyang.academia.edu/PiotrKonieczny
http://scholar.google.com/citations?user=gdV8_AEAAAAJ
http://en.wikipedia.org/wiki/User:Piotrus
On 7/30/2014 10:59, Lane Rasberry wrote:
Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
I am still thinking about what should be done with research.

yours,



On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:
Hi all,

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

Is RCom an official WMF body or a group of volunteers?

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

Is RCom a body regulating subject recruitment?

No, subject recruitment was only one among many areas of interest identified by its participants [2]

Is RCom still alive?

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

What about requests for private data?

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

How can we incentivize researchers to “give back” to the community?

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

Dario


On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--
Lane Rasberry
user:bluerasberry on Wikipedia
206.801.0814
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Heather Ford-3
In reply to this post by Dario Taraborelli-3
That is indeed really helpful, thanks for taking the time, Dario! 



On 29 July 2014 23:00, Dario Taraborelli <[hidden email]> wrote:
Hi all,

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

Is RCom an official WMF body or a group of volunteers?

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

Is RCom a body regulating subject recruitment?

No, subject recruitment was only one among many areas of interest identified by its participants [2]

Is RCom still alive?

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

What about requests for private data?

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

How can we incentivize researchers to “give back” to the community?

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

Dario


On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Kerry Raymond
In reply to this post by Lane Rasberry

30? No wonder we are worried about editor attrition J Seriously,

 

http://stats.wikimedia.org/EN/SummaryEN.htm

 

shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits per month) and 3K very active editors (>100 edits per month).

 

Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?

 

Kerry

 


From: [hidden email] [mailto:[hidden email]] On Behalf Of Lane Rasberry
Sent: Wednesday, 30 July 2014 12:00 PM
To: Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] discussion about wikipedia surveys

 

Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----

My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

 

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.

I am still thinking about what should be done with research.

yours,

 

 

On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:

Hi all,

 

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

 

Is RCom an official WMF body or a group of volunteers?

 

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

 

Is RCom a body regulating subject recruitment?

 

No, subject recruitment was only one among many areas of interest identified by its participants [2]

 

Is RCom still alive?

 

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

 

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

 

What about requests for private data?

 

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

 

How can we incentivize researchers to “give back” to the community?

 

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

 

Dario

 

 

On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:



Either [RCOM is] functioning or its not, surely?

 

Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

 

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

 

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

 

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

 

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

 

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

 

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

 

-Aaron

 

 

 

On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:

On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:

RCOM is not functioning as a complete group anymore.  

 

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?

 

However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

 

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

 

Best,

Heather.

 

 

-Aaron 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--

Lane Rasberry

user:bluerasberry on Wikipedia

206.801.0814
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Aaron Halfaker-2
For example, this proposal[1] was sent to me last year.  The researcher's plan was not to sample from the pool of active editors, but instead to contact the 500 most active Wikipedians on enwiki to survey them about their motivations.  There were several concerns raised about (1) whether the proposed study was duplicating prior work, (2) why the busiest editors needed to be surveyed and (3) whether the researcher's methodology would allow for the intended insights to be gained.   

Regretfully, the researcher decided not to respond to anyone other than myself and was also unwilling to work through many of the concerns I raised.


-Aaron


On Wed, Jul 30, 2014 at 8:19 PM, Kerry Raymond <[hidden email]> wrote:

30? No wonder we are worried about editor attrition J Seriously,

 

http://stats.wikimedia.org/EN/SummaryEN.htm

 

shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits per month) and 3K very active editors (>100 edits per month).

 

Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?

 

Kerry

 


From: [hidden email] [mailto:[hidden email]] On Behalf Of Lane Rasberry
Sent: Wednesday, 30 July 2014 12:00 PM


To: Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] discussion about wikipedia surveys

 

Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----

My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

 

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.

I am still thinking about what should be done with research.

yours,

 

 

On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:

Hi all,

 

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

 

Is RCom an official WMF body or a group of volunteers?

 

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

 

Is RCom a body regulating subject recruitment?

 

No, subject recruitment was only one among many areas of interest identified by its participants [2]

 

Is RCom still alive?

 

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

 

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

 

What about requests for private data?

 

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

 

How can we incentivize researchers to “give back” to the community?

 

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

 

Dario

 

 

On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:



Either [RCOM is] functioning or its not, surely?

 

Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

 

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

 

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

 

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

 

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

 

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

 

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

 

-Aaron

 

 

 

On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:

On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:

RCOM is not functioning as a complete group anymore.  

 

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?

 

However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

 

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

 

Best,

Heather.

 

 

-Aaron 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--

Lane Rasberry

user:bluerasberry on Wikipedia

<a href="tel:206.801.0814" value="+12068010814" target="_blank">206.801.0814
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Kerry Raymond

Well, it seems they took some of your advice. It did say “top 500 contributors” in its original form, but now says “any interested Wikipedia user”.

 

Kerry

 

 


From: Aaron Halfaker [mailto:[hidden email]]
Sent: Thursday, 31 July 2014 10:45 PM
To: Kerry Raymond; Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] discussion about wikipedia surveys

 

For example, this proposal[1] was sent to me last year.  The researcher's plan was not to sample from the pool of active editors, but instead to contact the 500 most active Wikipedians on enwiki to survey them about their motivations.  There were several concerns raised about (1) whether the proposed study was duplicating prior work, (2) why the busiest editors needed to be surveyed and (3) whether the researcher's methodology would allow for the intended insights to be gained.   

 

Regretfully, the researcher decided not to respond to anyone other than myself and was also unwilling to work through many of the concerns I raised.

 

 

-Aaron

 

On Wed, Jul 30, 2014 at 8:19 PM, Kerry Raymond <[hidden email]> wrote:

30? No wonder we are worried about editor attrition J Seriously,

 

http://stats.wikimedia.org/EN/SummaryEN.htm

 

shows that in May 2014 on en.WP we had about 32K active editors (> 5 edits per month) and 3K very active editors (>100 edits per month).

 

Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?

 

Kerry

 


From: [hidden email] [mailto:[hidden email]] On Behalf Of Lane Rasberry
Sent: Wednesday, 30 July 2014 12:00 PM


To: Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] discussion about wikipedia surveys

 

Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----

My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

 

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.

I am still thinking about what should be done with research.

yours,

 

 

On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:

Hi all,

 

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

 

Is RCom an official WMF body or a group of volunteers?

 

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

 

Is RCom a body regulating subject recruitment?

 

No, subject recruitment was only one among many areas of interest identified by its participants [2]

 

Is RCom still alive?

 

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

 

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

 

What about requests for private data?

 

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

 

How can we incentivize researchers to “give back” to the community?

 

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

 

Dario

 

 

On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

 

Either [RCOM is] functioning or its not, surely?

 

Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

 

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

 

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

 

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

 

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

 

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

 

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

 

-Aaron

 

 

 

On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:

On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:

RCOM is not functioning as a complete group anymore.  

 

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?

 

However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

 

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

 

Best,

Heather.

 

 

-Aaron 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--

Lane Rasberry

user:bluerasberry on Wikipedia

<a href="tel:206.801.0814" target="_blank" value="+12068010814">206.801.0814
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Lane Rasberry
In reply to this post by Piotr Konieczny-2

@Piotr
>Lane, how many survey requests do you get per year? And how much time do you spend on them?
>
Perhaps I got 10 survey requests, and perhaps 10 interview requests in the past year which were not obviously related to my outreach work or which I sought to have. I work in Wikipedia outreach and talk to a lot of people in that capacity too. I make myself visible.

I rarely complete surveys, and maybe have only done 5 in the last year with most or all of those being ones that I found rather than that found me.

I am contacted for being a certain kind of Wikipedian, and not for being any Wikipedian. People ask me about medicine, LGBT topics, paid editing or Wikipedian in Residence things. Unusually, I give my phone number and Skype out to a lot of people and do more Wikipedia discussion by voice than on-wiki or with typing. I do not know any other Wikipedian who says this.

>Those who don't want, don't take part in the survey. where's that disruption? What am I missing?
>
There is a historical precedent in human subject research which says that researchers cannot depend on their subjects to protect themselves, because this is presumed to always lead to unacceptable risk of harm to the subjects. Just saying that people can refuse surveys is out of bounds of contemporary research in western cultures. Some third party oversight from somewhere is necessary. I have no opinion about whether the Wikipedia community or WMF could provide that, but my initial thought is that if researchers follow their own institution's guidelines then things should be okay.

The most usual disruption is that the pool of researchers is large in comparison to the pool of people who could respond to surveys and interviews. Practically all researchers assume that it does not disrupt anything to ask, but there is a lot of asking and it is not obvious that Wikimedia community infrastructure should be used to serve people who are using it in ways that might be disturbing advancement of the Wikimedia mission.

@Kerry

32K active editors (> 5 edits per month) 3K very active editors (>100 edits per month).

 >Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?


No, you have it exactly! It is not the "Wikipedia for 10+ years with 10M edits or" that matters, but rather it is making any further distinction. A researcher who wants any demographic, like women, gay people, an ethnic minority, editors in a certain topic area, people who do a certain function like AfD, people who have had a certain problem, or many other things will cut the pool of 3000 down to 300. I would love for researchers to research those people doing 5-100 edits a month, but there is no way to reach this group and researchers rarely or never are interested in this group. Those 3000 are the ones I want to have protection because they are not a single group, but rather are lots of small groups each serving an important role. A typical survey is relevant to at most 10% of that 3000, meaning the potential research pool is rarely above 300 people. If the interviewer is imagining a research pool of 10,000 or more - which is the usual case - then they would be bold in trying to recruit and take time even though only 300 people could possibly respond. If they get 10% of the possible pool, which is an amazing response, then that could mean 30 responses and not enough to have valid results if the researcher hoped for 1% of 10,000. Also, when only 300 people do a task any time away from that is valuable time lost, and if researchers expected a larger pool, they may not be so careful with the volunteer's time. As researchers almost always come to Wikipedia completely as community outsiders, they almost always undervalue volunteer time here.




On Tue, Jul 29, 2014 at 10:39 PM, Piotr Konieczny <[hidden email]> wrote:
Lane, how many survey requests do you get per year? And how much time do you spend on them?

Because myself, being in the Top 100 most active editors and thus I'd think fitting in your group of " about 30 super busy people", I get about ~2 requests per year and they cost me few minutes at most, which even being "super busy" I find I can afford.

What I'd focus with a call for the researchers (perhaps another idea for best practices) would be to ask people to do proper lit review. I don't think we have too many surveys, but I do think we have a not-too-small percentage of them pointlessly replicating prior research (as in - we probably don't need a n-th paper on Wikipedian's motivations that badly...). Of course, people who can't be bothered to to a proper lit review can't probably be bothered to find out about our best practices guides, even if we clean the mess that our research pages are currently, so... :/
On 7/30/2014 10:59, Lane Rasberry wrote:
Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----
My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.
I am still thinking about what should be done with research.

yours,



On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:
Hi all,

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

Is RCom an official WMF body or a group of volunteers?

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

Is RCom a body regulating subject recruitment?

No, subject recruitment was only one among many areas of interest identified by its participants [2]

Is RCom still alive?

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

What about requests for private data?

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

How can we incentivize researchers to “give back” to the community?

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

Dario


On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:

Either [RCOM is] functioning or its not, surely?
 
Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

-Aaron




On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:
On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:
RCOM is not functioning as a complete group anymore.  

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?
 
However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.
 
There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

Best,
Heather.


-Aaron 
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--
Lane Rasberry
user:bluerasberry on Wikipedia
<a href="tel:206.801.0814" value="+12068010814" target="_blank">206.801.0814
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--
Lane Rasberry
user:bluerasberry on Wikipedia
206.801.0814
[hidden email]

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: discussion about wikipedia surveys

Kerry Raymond

I agree that researchers do not value the time of their subjects. But Wikipedia does not own the time of its volunteers; the volunteers choose to give it to Wikipedia and they may choose to give it to surveys and they may choose to watch Game of Thrones. Wikipedia can’t lose what it did not own; it can only be grateful for what is freely given. Ditto for researchers.

 

As I have previously stated, Wikipedia is within its rights to control the use of its communication channels in relation to promoting research surveys, because it does own those resources.

 

Kerry

 

 


From: [hidden email] [mailto:[hidden email]] On Behalf Of Lane Rasberry
Sent: Saturday, 2 August 2014 12:41 AM
To: Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] discussion about wikipedia surveys

 

 

@Piotr

>Lane, how many survey requests do you get per year? And how much time do you spend on them?

> 

Perhaps I got 10 survey requests, and perhaps 10 interview requests in the past year which were not obviously related to my outreach work or which I sought to have. I work in Wikipedia outreach and talk to a lot of people in that capacity too. I make myself visible.

 

I rarely complete surveys, and maybe have only done 5 in the last year with most or all of those being ones that I found rather than that found me.

 

I am contacted for being a certain kind of Wikipedian, and not for being any Wikipedian. People ask me about medicine, LGBT topics, paid editing or Wikipedian in Residence things. Unusually, I give my phone number and Skype out to a lot of people and do more Wikipedia discussion by voice than on-wiki or with typing. I do not know any other Wikipedian who says this.

 

>Those who don't want, don't take part in the survey. where's that disruption? What am I missing?

> 

There is a historical precedent in human subject research which says that researchers cannot depend on their subjects to protect themselves, because this is presumed to always lead to unacceptable risk of harm to the subjects. Just saying that people can refuse surveys is out of bounds of contemporary research in western cultures. Some third party oversight from somewhere is necessary. I have no opinion about whether the Wikipedia community or WMF could provide that, but my initial thought is that if researchers follow their own institution's guidelines then things should be okay.

 

The most usual disruption is that the pool of researchers is large in comparison to the pool of people who could respond to surveys and interviews. Practically all researchers assume that it does not disrupt anything to ask, but there is a lot of asking and it is not obvious that Wikimedia community infrastructure should be used to serve people who are using it in ways that might be disturbing advancement of the Wikimedia mission.

 

@Kerry

> 32K active editors (> 5 edits per month) 3K very active editors (>100 edits per month).

 >Or have I missed something here? Are researchers only interested in people who have been on Wikipedia for 10+ years with 10M edits or …?

 

No, you have it exactly! It is not the "Wikipedia for 10+ years with 10M edits or" that matters, but rather it is making any further distinction. A researcher who wants any demographic, like women, gay people, an ethnic minority, editors in a certain topic area, people who do a certain function like AfD, people who have had a certain problem, or many other things will cut the pool of 3000 down to 300. I would love for researchers to research those people doing 5-100 edits a month, but there is no way to reach this group and researchers rarely or never are interested in this group. Those 3000 are the ones I want to have protection because they are not a single group, but rather are lots of small groups each serving an important role. A typical survey is relevant to at most 10% of that 3000, meaning the potential research pool is rarely above 300 people. If the interviewer is imagining a research pool of 10,000 or more - which is the usual case - then they would be bold in trying to recruit and take time even though only 300 people could possibly respond. If they get 10% of the possible pool, which is an amazing response, then that could mean 30 responses and not enough to have valid results if the researcher hoped for 1% of 10,000. Also, when only 300 people do a task any time away from that is valuable time lost, and if researchers expected a larger pool, they may not be so careful with the volunteer's time. As researchers almost always come to Wikipedia completely as community outsiders, they almost always undervalue volunteer time here.

 

 

 

On Tue, Jul 29, 2014 at 10:39 PM, Piotr Konieczny <[hidden email]> wrote:

Lane, how many survey requests do you get per year? And how much time do you spend on them?

Because myself, being in the Top 100 most active editors and thus I'd think fitting in your group of " about 30 super busy people", I get about ~2 requests per year and they cost me few minutes at most, which even being "super busy" I find I can afford.

What I'd focus with a call for the researchers (perhaps another idea for best practices) would be to ask people to do proper lit review. I don't think we have too many surveys, but I do think we have a not-too-small percentage of them pointlessly replicating prior research (as in - we probably don't need a n-th paper on Wikipedian's motivations that badly...). Of course, people who can't be bothered to to a proper lit review can't probably be bothered to find out about our best practices guides, even if we clean the mess that our research pages are currently, so... :/

On 7/30/2014 10:59, Lane Rasberry wrote:

Hey guys,

I posted some thoughts to my own blog and am linking to those posts below. Everything I say on my blog is captured in the summary below, so feel free to not click through.

----

My biggest worry is that researchers who recruit human subjects assume that there are huge numbers of Wikipedians for them to survey, and consequently, they do not need to do a lot of advance survey preparation because there is no harm from distracting Wikipedians from their usual volunteer work. This assumption is wrong because actually almost every researcher recruiting human subjects wants Wikipedians who are in very short supply. Consequently, researchers do cause harm to the community by soliciting for volunteer time, and Wikipedia community benefit is dubious when researchers do not do sufficient preparation for their work. This is not quite accurate, but if there were one message I could convey to researchers, it would be "Your research participant pool only consists of about 30 super busy people and many other volunteers greatly depend on getting their time. When you take time from a Wikipedian, you are taking that time away from other volunteers who really need it, so be respectful of your intervention in our communities."
----

 

I do not want a lot of gatekeeping between researchers and the Wikipedia community, but at the same time, researchers should take professional pride in their work and take care not to disrupt Wikimedia community activities.

I am still thinking about what should be done with research.

yours,

 

 

On Tue, Jul 29, 2014 at 6:00 PM, Dario Taraborelli <[hidden email]> wrote:

Hi all,

 

I am a bit late in the game, but since so many questions were raised about RCom, its scope, its goals, the source of its authority etc. and I helped coordinate it in the early days I thought I’d chime in to clear some confusion. 

 

Is RCom an official WMF body or a group of volunteers?

 

RCom was created as a volunteer body to help design policies and best practices around research on Wikimedia projects. People who joined the committee did so on a volunteer basis and with a variety of interests by responding to a call for participation issued by WMF. Despite the fact that the original initiative came from WMF, its membership almost entirely consisted of non-WMF researchers and community members (those of us who are now with Wikimedia had no affiliation with the Foundation when RCom was launched [1]). RCom work was and remains 100% volunteer-driven, even for those of us who are full-time employees of the Foundation.

 

Is RCom a body regulating subject recruitment?

 

No, subject recruitment was only one among many areas of interest identified by its participants [2]

 

Is RCom still alive?

 

RCom stopped working a while ago as a group meeting on a regular basis to discuss joint initiatives. However, it spawned a large number of initiatives and workgroups that are still alive and kicking, some of which have evolved into other projects that are now only loosely associated with RCom. These include reviewing subject recruitment requests, but also the Research Newsletter, which has been published monthly for the last 3 years; countless initiatives in the area of open access; initiatives to facilitate Wikimedia data documentation and data discoverability; hackathons and outreach events aimed at bringing together researchers and Wikimedia contributors. Subject recruitment reviews and discussions are still happening, and I believe they provide a valuable service when you consider that they are entirely run by a microscopic number of volunteers. I don’t think that the alternative between “either RCom exists and it functions effectively or reviews should immediately stop” is well framed or even desirable, for the reasons that I explain below. 

 

What’s the source of RCom’s authority in reviewing subject recruitment requests?

 

Despite the perception that one of RCom’s duties would be to provide formal approval for research projects, it was never designed to do so and it never had the power to enforce formal review decisions. Instead, it was offered as a volunteer support service in an effort to help minimize disruption, improve the relevance of research involving Wikimedia contributors, sanity check the credentials of the researchers, create collaborations between researchers working on the same topic. The lack of community or WMF policies to back subject recruitment caused in the past few years quite some headaches, particularly in those cases in which recruitment attempts were blocked and referred to the RCom in order to “obtain formal approval”. The review process itself was meant to be as inclusive as possible and not restricted to RCom participants and researchers having their proposal reviewed were explicitly invited to address any questions or concerns raised by community members on the talk page. I totally agree that the way in which the project templates and forms were designed needs some serious overhaul to remove any indication of a binding review process or a commitment for reviews to be delivered within a fixed time frame. I cannot think of any example in which the review process discriminated some type of projects (say qualitative research) in favor of other types of research, but I am sure different research proposals attracted different levels of participation and interest in the review process. My recommendation to anyone interested in designing future subject recruitment processes is to focus on a lightweight review process open to the largest possible number of community members but backed by transparent and enforceable policies. It’s a really hard problem and there is simply no obvious silver bullet solution that can be found without some experimentation and fault tolerance.

 

What about requests for private data?

 

Private data and technical support requests from WMF are a different story: they were folded into the list of frequently asked questions hosted on the RCom section of Meta, but by definition they require a direct and substantial involvement from the Foundation: (1) they involve WMF as the legal entity that would be held liable for disclosing data in breech of its privacy policies and (2) they involve paid staff resources and need to be prioritized against a lot of other requests. There are now dedicated sections on private data on the Wikimedia Privacy Policy [4] and Data Retention guidelines [5]. Many people, including myself and other members of the Foundation’s Analytics team, believe that we should try and collect the minimum amount of private data that we need in order to operate and study our projects and make all those types of aggregate/sanitized data that we can retain indefinitely publicly available to everyone under open licenses. We’ve already started a process to do so and to ensure that more data (for example, data collected via site instrumentation [6]) be exposed via Labs or other APIs, in the respect of our users’ privacy.

 

How can we incentivize researchers to “give back” to the community?

 

In the early days we drafted a set of requirements [3] to make sure we could get back as much as possible from research involving WMF resources. It’s been hard to implement these requirements without policies to enforce them. The suggestion of having more researchers apply for a slot at the Research Showcase to present their work is an excellent idea that we should consider. In general, the Research team at WMF is always interested in hearing about incentives to drive more interest towards actionable research on Wikimedia projects.

 

Dario

 

 

On Jul 29, 2014, at 6:49 PM, Aaron Halfaker <[hidden email]> wrote:



Either [RCOM is] functioning or its not, surely?

 

Well, I explained that there are functioning sub-committees still.  In other words, there are initiatives that RCOM started that are alive and successful, but we no longer coordinate as a larger group.  I don't know how else to explain it.  I guess you could say that RCOM is still functioning and that we no longer require/engage in group meetings. 

 

As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

That's a proposed policy.  Until it is passed by consensus, the "must" is a proposed term.  I think that it should be "must", but until that consensus is reached, I'll continue to say that it "ought to".  

 

Regarding researchers stating what should be regulated, I think there is a big difference between deciding what should be regulated and being involved in the discussion of *how* it should be regulated.  Hence why I welcome participation.  What I'm saying is that you have a vested interest in not being regulated, but I'd still like to discuss how your activities can be regulated effectively & efficiently.  Does that make sense? 

 

 b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

I welcome you to raise it to them.  I don't think it is worth their time, but they might disagree.  

 

But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I think that the proposed policy on English Wikipedia does that quite well.  That's why I directed people there.  Also, again, I am not working on RCOM or subject recruitment as a WMF employee.  I do this in my volunteer time.  This is true of all of RCOM who happen to also be staff.  

 

if you want process to be more clearly documented, you also have to address people like Poitr who would rather not have processes described in detail.  When you guys work out how clearly you want a process to be described, please let me know.  I'm tired of re-spec'ing processes.  This is the third iteration.  

 

If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion.

 

Heather, that is a proposed policy page on English Wikipedia.  It is not part of RCOM.  It would render RCOM irrelevant for subject recruitment concerns.  That's why I started it.  I don't think that RCOM/WMF/researchers should own subject recruitment review.  I think the community being studied should own it and that RCOM/WMF/researchers should participate.

 

Also, I am not your employee.  This is my volunteer time.  I don't have much of it, so I focus on keeping the system running -- and it is -- and improving the system -- which is the proposal I linked to.  If you want something done and other volunteers don't have time to do it.  Do it yourself.  

 

-Aaron

 

 

 

On Tue, Jul 29, 2014 at 10:23 AM, Heather Ford <[hidden email]> wrote:

On Tue, Jul 29, 2014 at 10:25 AM, Aaron Halfaker <[hidden email]> wrote:

RCOM is not functioning as a complete group anymore.  

 

I'm a little confused why this wasn't made clear right at the beginning of this thread e.g. when others suggested this might be the case and you refuted them? Also, I'm not sure what 'functioning as a complete group' actually means. Either its functioning or its not, surely?

 

However, we split into sub-committees while we were still a functioning group.  The subject recruitment sub-committee and newsletter sub-committees are performing vital functions still. 

 

I never stated that research recruiting needs RCOM approval. I definitely said that it "ought to" have RCOM approval.

 

So, does that mean that is what the policy *ought to* be now? And do you believe that this should this be the way that the policy gets decided? Because it isn't right now as far as I can see. As Kerry noted earlier on, the policy as it stands [1] says that researchers "must" obtain approval through the process described. If the wording now needs to be changed to "ought to" then surely this requires more consensus than your single message here? 

 

re. the comment that I (and the other researchers?) on this list shouldn't be the ones to decide what the regulation should be, I disagree on two counts. a) It seems on the one hand that you want this to be "self-regulation" i.e. you invited researchers on this list to join R-COM at the beginning of this thread, but that you don't think that the researchers here should be able to determine what to regulate. I know that you're looking for an inclusive process but you can't have it both ways: if we are going to help regulate, then we need to at least help decide how to regulate. b) Pine suggested a board decision on this earlier one to obtain clarity and I supported this but it was met with silence, which is why I followed up.

 

There are also more than two "review coordinators" (not not "reviewers") -- it's just that DarTar and I have accepted the burden of distributing work.  When people are busy, we often coordinate the reviews ourselves. 

 

I can understand your frustration; I really can! I know that you've done a lot of really great, prior work on this and I don't think any of us are saying that we need to throw the baby out with the bathwater. But what is clear is that clarification is required - especially on the distribution of tasks between Foundation employees, the research community and Wikimedia editors. And this is *especially* true for people outside this list.

 

I welcome your edits to make it clear that review is optional.  As you might imagine, I have plenty of work to do and I appreciate your good-faith collaboration on improving our research documentation. 

 

I'm frustrated by this response. If the policy is incorrectly described on the policy pages, then someone from RCom (or whatever it is now called) should be the one to change this - preferably with some discussion. I find it frustrating that WMF employees are often the ones who make the final policy pronouncements but then tell others to implement it. And if we don't do the work, then we're apparently not assuming good faith. 

 

This is a great opportunity to rejuvenate the process; hopefully it will eventually be seen that way :)

 

Best,

Heather.

 

 

-Aaron 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--

Lane Rasberry

user:bluerasberry on Wikipedia

<a href="tel:206.801.0814" target="_blank" value="+12068010814">206.801.0814
[hidden email]



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



 

--

Lane Rasberry

user:bluerasberry on Wikipedia

206.801.0814
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
123