soliciting participants for Wikipedia studies

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

soliciting participants for Wikipedia studies

Phoebe Ayers-2
Hello all,

This is sort of related to the previous thread on consent, and is
something that's been on my mind for a while.

As someone who is reasonably visible in the wiki research community
and Wikipedia, I get asked fairly often to either a) help recruit
participants for studies of Wikipedia, or b) participate myself in
such studies.

The common theme of such requests is often:
* The researcher wants to find people who are invested in Wikipedia,
for qualitative studies of contributors
* The researcher does not know how to go about doing this
* What "invested" means is often poorly defined, since the researcher
is often trying to figure out what participation looks like more
generally
* The researcher has done the standard things (posted on the mailing
list, on the village pump) and hasn't gotten any results; or has
semi-randomly posted on people's talk pages, potentially getting a
warning about spamming in the process

As a result:
* many of the same people (i.e. very visible contributors) keep
getting asked to participate in different studies; or
* the researcher is left with a self-selected group of people from the
mailing lists or other places, which may in no way represent 'the
community' (my hypothesis is that we have many small communities,
working under the greater umbrella of Wikipedia); and who may be
people who are particularly outspoken or disgruntled; or
* the researcher does not get enough participants to do a good study

So:
* Is there a good solution for these problems?
* Can we come up with "best practices" or advice for people who are
trying to recruit Wikipedians for studies?
* What about some sort of infrastructure or wikiproject to support
these requests? Every time I get one of these emails I would really
like to pass it on to a group of people to deal with, but I am not
sure who, and this mailing list seems too small and focused to support
such requests.

best,
Phoebe

p.s. I think "any wiki with a large base of contributors" could be
substituted for "Wikipedia" here -- this is probably a problem with
studying any large community-run site. But most of my requests have
come from people specifically interested in Wikipedia.

- phoebe s. ayers | [hidden email]

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Piotr Konieczny-2
phoebe ayers wrote:

> * The researcher has done the standard things (posted on the mailing
> list, on the village pump) and hasn't gotten any results; or has
> semi-randomly posted on people's talk pages, potentially getting a
> warning about spamming in the process
>
> As a result:
> * many of the same people (i.e. very visible contributors) keep
> getting asked to participate in different studies; or
> * the researcher is left with a self-selected group of people from the
> mailing lists or other places, which may in no way represent 'the
> community' (my hypothesis is that we have many small communities,
> working under the greater umbrella of Wikipedia); and who may be
> people who are particularly outspoken or disgruntled; or

There is an interesting tool:
http://en.wikipedia.org/wiki/Special:Random/User
If anybody wants a truly random sample of Wikipedia users, that's a good
way to do it.

Is asking for a survey spamming? That's a good question. If we could
raise it on a community page and get a consensus that it is not, than we
could potentially create a bot that could be fed a survey and would
deliver it to x random users via the above page.

--
Piotr Konieczny

"The problem about Wikipedia is, that it just works in reality, not in
theory."

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Gregory Maxwell
On Wed, Nov 19, 2008 at 2:54 PM, Piotr Konieczny <[hidden email]> wrote:
> There is an interesting tool:
> http://en.wikipedia.org/wiki/Special:Random/User
> If anybody wants a truly random sample of Wikipedia users, that's a good
> way to do it.

Thats more than a slightly tedious process: The overwhelming majority
of user accounts on the Enwp (and to a lesser extent on other large
projects) are inactive/have no-non-deleted etc.

If a researcher wants a constrained sample of users such as "Users who
have made more than 1 edit and who have edited in the past month" or
whatever, they can ask anyone with a toolserver account to make them
such a list.

> Is asking for a survey spamming? That's a good question. If we could
> raise it on a community page and get a consensus that it is not, than we
> could potentially create a bot that could be fed a survey and would
> deliver it to x random users via the above page.

It might be useful to create a way for users to opt-out in bulk, I
doubt many would do it, but it might help keep the few who would from
becoming upset and going on a crusade after the "dreadful spammer".

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Phoebe Ayers-2
In reply to this post by Piotr Konieczny-2
On Wed, Nov 19, 2008 at 11:54 AM, Piotr Konieczny <[hidden email]> wrote:

> phoebe ayers wrote:
>
>> * The researcher has done the standard things (posted on the mailing
>> list, on the village pump) and hasn't gotten any results; or has
>> semi-randomly posted on people's talk pages, potentially getting a
>> warning about spamming in the process
>>
>> As a result:
>> * many of the same people (i.e. very visible contributors) keep
>> getting asked to participate in different studies; or
>> * the researcher is left with a self-selected group of people from the
>> mailing lists or other places, which may in no way represent 'the
>> community' (my hypothesis is that we have many small communities,
>> working under the greater umbrella of Wikipedia); and who may be
>> people who are particularly outspoken or disgruntled; or
>
> There is an interesting tool:
> http://en.wikipedia.org/wiki/Special:Random/User
> If anybody wants a truly random sample of Wikipedia users, that's a good
> way to do it.
>
> Is asking for a survey spamming? That's a good question. If we could
> raise it on a community page and get a consensus that it is not, than we
> could potentially create a bot that could be fed a survey and would
> deliver it to x random users via the above page.

This seems like an interesting idea for random studies.

Many of the studies I have encountered are looking for "some
experienced wikipedians" to participate, and that seems like a much
harder thing to pull off. The chances of getting an "experienced" user
who is willing to talk to you out of the 8 million registered accounts
on en:wp seems low if you go about it by trying to randomly poll
users. I think a lot of researchers want people who are 'experienced'
by some metric so they can give informed feedback on processes,
policies, social dynamics, interfaces... but it's also pretty common
that most contributors focus in on only a few areas. So how to get
experienced people who can speak to the questions you are interested
in, particularly if you're not a wikipedian yourself?

-- phoebe

p.s. This message is not a comment on Avanidhar's study or Stuart's
study or any other particular studies... rather, it's something I've
been thinking about for a long time as an issue for everyone studying
Wikipedia.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

avani-3
Dear  All,

Thanks for bringing this topic up on the list. True, we do seek random
users from the Wikipedia community for our studies. But, like phoebe
mentioned, we usually expect a particular trait/quality in the users.
Generally, most studies are aimed at "experienced editors": those who have
been along for a while, who understand wikipedia policies and who make
significant contributions to any article/set of articles over a period of
time.

It would be great if there were some mechanism to invite editors to
participate in our studies without giving an image of us being "spammers"
or "vandals". After all, all we need is a sizeable number of participants
out of millions of editors, which would not only make our research
credible, but also worth mentioning while contemplating future directions
for any Wiki based system.

~ Avanidhar

On Nov 19 2008, phoebe ayers wrote:

>On Wed, Nov 19, 2008 at 11:54 AM, Piotr Konieczny <[hidden email]> wrote:
>> phoebe ayers wrote:
>>
>>> * The researcher has done the standard things (posted on the mailing
>>> list, on the village pump) and hasn't gotten any results; or has
>>> semi-randomly posted on people's talk pages, potentially getting a
>>> warning about spamming in the process
>>>
>>> As a result:
>>> * many of the same people (i.e. very visible contributors) keep
>>> getting asked to participate in different studies; or
>>> * the researcher is left with a self-selected group of people from the
>>> mailing lists or other places, which may in no way represent 'the
>>> community' (my hypothesis is that we have many small communities,
>>> working under the greater umbrella of Wikipedia); and who may be
>>> people who are particularly outspoken or disgruntled; or
>>
>> There is an interesting tool:
>> http://en.wikipedia.org/wiki/Special:Random/User
>> If anybody wants a truly random sample of Wikipedia users, that's a good
>> way to do it.
>>
>> Is asking for a survey spamming? That's a good question. If we could
>> raise it on a community page and get a consensus that it is not, than we
>> could potentially create a bot that could be fed a survey and would
>> deliver it to x random users via the above page.
>
>This seems like an interesting idea for random studies.
>
>Many of the studies I have encountered are looking for "some
>experienced wikipedians" to participate, and that seems like a much
>harder thing to pull off. The chances of getting an "experienced" user
>who is willing to talk to you out of the 8 million registered accounts
>on en:wp seems low if you go about it by trying to randomly poll
>users. I think a lot of researchers want people who are 'experienced'
>by some metric so they can give informed feedback on processes,
>policies, social dynamics, interfaces... but it's also pretty common
>that most contributors focus in on only a few areas. So how to get
>experienced people who can speak to the questions you are interested
>in, particularly if you're not a wikipedian yourself?
>
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Gorbatai, Andreea
Hi all -

I am relatively new to the list but I have been looking at Wikipedia (quant) data for a few years now. I recently started contacting people for interviews regarding socialization/social networks of collaboration, so I second your concerns. It's hard to contact users who have left and even harder to get a response from inexperienced users, which I would love to know more about in order to understand the process through which people gain experience/ make sense of their Wikipedia participation.

Is there a way to do this through the Wikimedia Foundation,  any thoughts?

Thank you everyone for being so willing to help (each other), this list is such a wonderful resource!

Regards,
Andreea

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Wednesday, November 19, 2008 4:40 PM
To: Research into Wikimedia content and communities
Subject: Re: [Wiki-research-l] soliciting participants for Wikipedia studies

Dear  All,

Thanks for bringing this topic up on the list. True, we do seek random
users from the Wikipedia community for our studies. But, like phoebe
mentioned, we usually expect a particular trait/quality in the users.
Generally, most studies are aimed at "experienced editors": those who have
been along for a while, who understand wikipedia policies and who make
significant contributions to any article/set of articles over a period of
time.

It would be great if there were some mechanism to invite editors to
participate in our studies without giving an image of us being "spammers"
or "vandals". After all, all we need is a sizeable number of participants
out of millions of editors, which would not only make our research
credible, but also worth mentioning while contemplating future directions
for any Wiki based system.

~ Avanidhar

On Nov 19 2008, phoebe ayers wrote:

>On Wed, Nov 19, 2008 at 11:54 AM, Piotr Konieczny <[hidden email]> wrote:
>> phoebe ayers wrote:
>>
>>> * The researcher has done the standard things (posted on the mailing
>>> list, on the village pump) and hasn't gotten any results; or has
>>> semi-randomly posted on people's talk pages, potentially getting a
>>> warning about spamming in the process
>>>
>>> As a result:
>>> * many of the same people (i.e. very visible contributors) keep
>>> getting asked to participate in different studies; or
>>> * the researcher is left with a self-selected group of people from the
>>> mailing lists or other places, which may in no way represent 'the
>>> community' (my hypothesis is that we have many small communities,
>>> working under the greater umbrella of Wikipedia); and who may be
>>> people who are particularly outspoken or disgruntled; or
>>
>> There is an interesting tool:
>> http://en.wikipedia.org/wiki/Special:Random/User
>> If anybody wants a truly random sample of Wikipedia users, that's a good
>> way to do it.
>>
>> Is asking for a survey spamming? That's a good question. If we could
>> raise it on a community page and get a consensus that it is not, than we
>> could potentially create a bot that could be fed a survey and would
>> deliver it to x random users via the above page.
>
>This seems like an interesting idea for random studies.
>
>Many of the studies I have encountered are looking for "some
>experienced wikipedians" to participate, and that seems like a much
>harder thing to pull off. The chances of getting an "experienced" user
>who is willing to talk to you out of the 8 million registered accounts
>on en:wp seems low if you go about it by trying to randomly poll
>users. I think a lot of researchers want people who are 'experienced'
>by some metric so they can give informed feedback on processes,
>policies, social dynamics, interfaces... but it's also pretty common
>that most contributors focus in on only a few areas. So how to get
>experienced people who can speak to the questions you are interested
>in, particularly if you're not a wikipedian yourself?
>
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Joseph Reagle
On Wednesday 19 November 2008, Gorbatai, Andreea wrote:
> I am relatively new to the list but I have been looking at Wikipedia (quant) data for a few years now. I recently started contacting people for interviews regarding socialization/social networks of collaboration, so I second your concerns. It's hard to contact users who have left and even harder to get a response from inexperienced users, which I would love to know more about in order to understand the process through which people gain experience/ make sense of their Wikipedia participation.

I'll just note that my one brief experience in soliciting interviews online was rather troubled (see below). Fortunately, for practical and theoretical reasons I preferred making use of public practice and discussion. In any case, should I need to do so again, the best "interviews" I did make were by going to F2F meetings and connecting with Wikipedians. This isn't a random or representative sample of course.

[[ http://reagle.org/joseph/2005/ethno/leadership.html

On a suggestion, I developed a brief questionnaire to engage with editors of the Harry Potter Project pages but, as expected, received few responses. Open content communities are, presently, often studied (with similar questionnaires) and participants might have little interest in taking time away from their actual (volunteer) work to respond to yet another. (As a participant, I have never responded to such a questionnaire.) Contacting actual participants can be difficult as well, as Lorenzon (2005) noted: "Many editors have their own user page which give information about them but few give out their real names and contact information." I made my solicitation on the Talk page for the Project as well as the Talk pages of a handful of prominent editors, without much success. Additionally, because most all the discourse is public and the community is otherwise so reflective, there is an abundance of existing data situated in actual practice. This is not to say such research discussions are not useful; once I developed my questions I was interested in receiving answers and the single response was informative. Fortunately, while responses to questionnaires can be hard to obtain, I also do not think them necessary to understand this community. Instead, one must follow (or even engage) in the practice: "A culture is expressed (or constituted) only by the actions and words of its members and must be interpreted by, not given to, a field worker" (Van Maanen 1988).

]]

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Reid Priedhorsky-2
In reply to this post by Piotr Konieczny-2
Piotr Konieczny wrote:
>
> Is asking for a survey spamming? That's a good question. If we could
> raise it on a community page and get a consensus that it is not, than we
> could potentially create a bot that could be fed a survey and would
> deliver it to x random users via the above page.

One of my colleagues was metaphorically eaten alive for posting requests
for research participation on user talk pages (a few dozen IIRC). In her
case the users in question were chosen fairly specifically (perhaps by
the kinds of pages they edited?), and I think she had 100-200 candidate
participants.

I've bcc'ed her in case she wants to chime in.

Let me voice my strong support for Phoebe's effort. I'm not sure what I
can do personally (though please enlighten me if you know), but some
process with community buy-in for soliciting random Wikipedia users for
research purposes would be extremely useful to the research community.

Reid



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
fn
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

fn



> "survey spamming"

In medical trials subjects are very often paid. I would guess that the
response rate for a Wikipedia survey would increase if the researcher
mentioned that the subjects/survey taker would be paid.

In a survey for our "Lyngby" programming toolbox we made a lottery among
the respondents and gave away a gift card for the Amazon online bookstore
to the winner. Among the 270 registered users we got 28 responses and one
regarding our survey as spam.


/Finn

___________________________________________________________________

          Finn Aarup Nielsen, DTU Informatics, Denmark
  Lundbeck Foundation Center for Integrated Molecular Brain Imaging
    http://www.imm.dtu.dk/~fn/      http://nru.dk/staff/fnielsen/
___________________________________________________________________


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: soliciting participants for Wikipedia studies

Piotr Konieczny-2
In reply to this post by Piotr Konieczny-2
Piotr Konieczny wrote:

> There is an interesting tool:
> http://en.wikipedia.org/wiki/Special:Random/User

http://en.wikipedia.org/wiki/Special:Random/User_talk is even better

Also, a thought on how to reduce spammer criticism: creata a list for
opt-outs that the survey bot would consider.

If we could indeed get advanced help from somebody on toolserv, who
could add options to the bot that would allow us to screen for things
like recent activity or min. number of edits, it would be great.

--
Piotr Konieczny

"The problem about Wikipedia is, that it just works in reality, not in
theory."

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l