Recognizing domain experts contribution to Wikipedia

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Recognizing domain experts contribution to Wikipedia

Alex Yarovoy
Hi All,

I'm a Master student working under the supervision of Drs. Arazy and Minkov
(Haifa U)
My research explores the extent to which  "recognized domain experts"
contribute to Wikipedia.
(I use a narrow definition for "recognized domain experts" to include those
with academic qualifications in the relevant topic).
I manually tracked these experts using a variety of sources, and then use
machine learning methods for automatically identifying domain experts
within Wikipedia editors.

I'm writing to explore whether this research is on interest to the
community and to learn if other people have already tackled this research
question.

Thank you in advance for pointing me to relevant research projects
Alex
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Leila Zia
Hi Alex,

On Fri, Jul 7, 2017 at 8:20 AM, Alex Yarovoy <[hidden email]> wrote:

> Hi All,
>
> I'm a Master student working under the supervision of Drs. Arazy and Minkov
> (Haifa U)
>

​This is great. :) Thanks in advance for the time and effort you will put
to help us improve (our understanding of) Wikimedia projects.​


> My research explores the extent to which  "recognized domain experts"
> contribute to Wikipedia.
> (I use a narrow definition for "recognized domain experts" to include those
> with academic qualifications in the relevant topic).
> I manually tracked these experts using a variety of sources, and then use
> machine learning methods for automatically identifying domain experts
> within Wikipedia editors.
>

​I'm not sure if this part of your research is done, if it's not, section
2.3.1. of https://arxiv.org/pdf/1604.03235.pdf can be of interest to you as
a starting point for building topical models based on past contributions.


> I'm writing to explore whether this research is on interest to the
> community and to learn if other people have already tackled this research
> question.
>

​I'm not speaking on behalf of the community but providing my personal view
which should be counted as one view:​ :)

​We don't have a good understanding of why experts (the way you define the
term) do or do not contribute to Wikipedia (or other Wikimedia projects).
Understanding the incentive mechanisms (and I emphasize on mechanisms) that
can encourage these people to contribute is very valuable. Understanding
the blockers for their contributions (which can be related to the incentive
problem, but not necessarily), is also very valuable. We also don't
understand how such incentives change across Wikipedia languages or
Wikimedia projects, if at all (For example, does a professional
photographer who uploads photos to Wikimedia Commons do this with the same
incentive as a doctor adding content about an epidemic on Hindi Wikipedia
or a university professor who is improving articles on convex optimization
in Hebrew?)

What I'd like to recommend is that you keep the practical component of the
research you will be doing in this space in mind. A good rule of thumb is:
if I answer question x, how will it help Wikimedians recruit for or
encourage such experts to join Wikimedia? This is a high bar to meet,
especially in a master thesis, but it's a good bar to keep in mind as you
think about research questions. :)

​Good luck with your research! :)

Best,
Leila

p.s. btw, do make sure you share the result of your research as you develop
it with the research community. We have, for example, Wiki Workshop which
happens at least once a year and we'd love to hear more about your research
there. (last year's page: http://wikiworkshop.org/2017/ ) We had Ofer as an
invited speaker to the workshop a couple of years ago. :)

--
Leila Zia
Senior Research Scientist
Wikimedia Foundation




> Thank you in advance for pointing me to relevant research projects
> Alex
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Pine W
In reply to this post by Alex Yarovoy
Hi Alex,

I believe that this is a subject of interest to the community. It would
indeed be helpful to know the percentage of people with graduate-level
academic qualifications who regularly make contributions on English
Wikipedia and other language editions of Wikipedia.

I'd suggest thinking about the following:

1. In general, academics don't receive benefits to their C.V. from
contributing to Wikipedia. My guess is that this is a major reason why
relatively few academics contribute to Wikipedia on a regular basis. It
might be interesting if you can produce data that confirms a hypothesis
like this.

2. I would encourage changing the term that you use from "recognized domain
experts" to "people with graduate-level academic qualifications". In the
U.S., in many domains, there are multiple ways for people to gain
reputations of being experts in domain; an academic qualification is often
not required, although it may be helpful.

Pine


On Fri, Jul 7, 2017 at 8:20 AM, Alex Yarovoy <[hidden email]> wrote:

> Hi All,
>
> I'm a Master student working under the supervision of Drs. Arazy and Minkov
> (Haifa U)
> My research explores the extent to which  "recognized domain experts"
> contribute to Wikipedia.
> (I use a narrow definition for "recognized domain experts" to include those
> with academic qualifications in the relevant topic).
> I manually tracked these experts using a variety of sources, and then use
> machine learning methods for automatically identifying domain experts
> within Wikipedia editors.
>
> I'm writing to explore whether this research is on interest to the
> community and to learn if other people have already tackled this research
> question.
>
> Thank you in advance for pointing me to relevant research projects
> Alex
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Stuart A. Yeates
It is worth remembering that via the orcid identifier in the authority
control template, the is now a standard linked-data mechanism for
researchers to identify themselves. I have no idea whether anyone is
looking at that though.

Cheers
Stuart

On Saturday, July 8, 2017, Pine W <[hidden email]> wrote:

> Hi Alex,
>
> I believe that this is a subject of interest to the community. It would
> indeed be helpful to know the percentage of people with graduate-level
> academic qualifications who regularly make contributions on English
> Wikipedia and other language editions of Wikipedia.
>
> I'd suggest thinking about the following:
>
> 1. In general, academics don't receive benefits to their C.V. from
> contributing to Wikipedia. My guess is that this is a major reason why
> relatively few academics contribute to Wikipedia on a regular basis. It
> might be interesting if you can produce data that confirms a hypothesis
> like this.
>
> 2. I would encourage changing the term that you use from "recognized domain
> experts" to "people with graduate-level academic qualifications". In the
> U.S., in many domains, there are multiple ways for people to gain
> reputations of being experts in domain; an academic qualification is often
> not required, although it may be helpful.
>
> Pine
>
>
> On Fri, Jul 7, 2017 at 8:20 AM, Alex Yarovoy <[hidden email]
> <javascript:;>> wrote:
>
> > Hi All,
> >
> > I'm a Master student working under the supervision of Drs. Arazy and
> Minkov
> > (Haifa U)
> > My research explores the extent to which  "recognized domain experts"
> > contribute to Wikipedia.
> > (I use a narrow definition for "recognized domain experts" to include
> those
> > with academic qualifications in the relevant topic).
> > I manually tracked these experts using a variety of sources, and then use
> > machine learning methods for automatically identifying domain experts
> > within Wikipedia editors.
> >
> > I'm writing to explore whether this research is on interest to the
> > community and to learn if other people have already tackled this research
> > question.
> >
> > Thank you in advance for pointing me to relevant research projects
> > Alex
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email] <javascript:;>
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email] <javascript:;>
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>


--
--
...let us be heard from red core to black sky
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Pine W
Stuart reminds me of a point that I forgot to mention. While I believe that
directly contributing to Wikipedia is rarely beneficial for academics' CVs,
having one's work cited on Wikipedia might be viewed positively. Perhaps
one way that academics might be incentivized to contribute more to
Wikipedia is by encouraging them to post references to their works on talk
pages when the academics think that their work could be beneficial to
articles.

Pine


On Sat, Jul 8, 2017 at 4:14 AM, Stuart A. Yeates <[hidden email]> wrote:

> It is worth remembering that via the orcid identifier in the authority
> control template, the is now a standard linked-data mechanism for
> researchers to identify themselves. I have no idea whether anyone is
> looking at that though.
>
> Cheers
> Stuart
>
> On Saturday, July 8, 2017, Pine W <[hidden email]> wrote:
>
> > Hi Alex,
> >
> > I believe that this is a subject of interest to the community. It would
> > indeed be helpful to know the percentage of people with graduate-level
> > academic qualifications who regularly make contributions on English
> > Wikipedia and other language editions of Wikipedia.
> >
> > I'd suggest thinking about the following:
> >
> > 1. In general, academics don't receive benefits to their C.V. from
> > contributing to Wikipedia. My guess is that this is a major reason why
> > relatively few academics contribute to Wikipedia on a regular basis. It
> > might be interesting if you can produce data that confirms a hypothesis
> > like this.
> >
> > 2. I would encourage changing the term that you use from "recognized
> domain
> > experts" to "people with graduate-level academic qualifications". In the
> > U.S., in many domains, there are multiple ways for people to gain
> > reputations of being experts in domain; an academic qualification is
> often
> > not required, although it may be helpful.
> >
> > Pine
> >
> >
> > On Fri, Jul 7, 2017 at 8:20 AM, Alex Yarovoy <[hidden email]
> > <javascript:;>> wrote:
> >
> > > Hi All,
> > >
> > > I'm a Master student working under the supervision of Drs. Arazy and
> > Minkov
> > > (Haifa U)
> > > My research explores the extent to which  "recognized domain experts"
> > > contribute to Wikipedia.
> > > (I use a narrow definition for "recognized domain experts" to include
> > those
> > > with academic qualifications in the relevant topic).
> > > I manually tracked these experts using a variety of sources, and then
> use
> > > machine learning methods for automatically identifying domain experts
> > > within Wikipedia editors.
> > >
> > > I'm writing to explore whether this research is on interest to the
> > > community and to learn if other people have already tackled this
> research
> > > question.
> > >
> > > Thank you in advance for pointing me to relevant research projects
> > > Alex
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email] <javascript:;>
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email] <javascript:;>
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
>
>
> --
> --
> ...let us be heard from red core to black sky
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Alex Yarovoy
In reply to this post by Alex Yarovoy
Thank you Leila, Stuart, Pine
We will follow up on these comments and pointers

A few additional words about this research -
Our narrow definition of formal expertise focuses on those with academic
qualifications who have published a scholarly work (i.e. appears in Google
Scholar) in the topic of the specific Wikipedia articles where one was
active.
We acknowledge that many experts do not have academic qualifications.
The choice of "formal" (i.e. academic in this context) expertise enabled a
concrete operationalization and measurement.
We welcome any ideas for pinpointing informal experts.

We are currently in the first phase of research where we try to identify
these formal experts. We've spent considerable amount of time in
identifying 500 such experts, and now we use machine learning techniques to
automatically spot them (preliminary results are quite good).
Once this is done, we can start asking interesting questions, such as:
- What is the relative role of these formal experts to overall content
contributed to Wikipedia?
- Are formal experts' contributions "better"? (e.g. survive longer or
result in increased quality score (per ORES)
- Who are those formal experts? anonymous contributors? registered users?
do they take additional roles within the community?
- Formal experts' motivation

Any other ideas for taking this research forward are more than welcome.

Thank you,
Ofer, Einat and Alex
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Shani Evenstein
Hi Alex,

Welcome to the community!

I am based at Tel Aviv University, where I teach 2 Wiki courses I developed
and reseach Wikipedia & Wikidata (among others). If there's anything I can
do to help, I'm a phone call away.  :-)

Best,
Shani.

On 10 Jul 2017 23:02, "Alex Yarovoy" <[hidden email]> wrote:

> Thank you Leila, Stuart, Pine
> We will follow up on these comments and pointers
>
> A few additional words about this research -
> Our narrow definition of formal expertise focuses on those with academic
> qualifications who have published a scholarly work (i.e. appears in Google
> Scholar) in the topic of the specific Wikipedia articles where one was
> active.
> We acknowledge that many experts do not have academic qualifications.
> The choice of "formal" (i.e. academic in this context) expertise enabled a
> concrete operationalization and measurement.
> We welcome any ideas for pinpointing informal experts.
>
> We are currently in the first phase of research where we try to identify
> these formal experts. We've spent considerable amount of time in
> identifying 500 such experts, and now we use machine learning techniques to
> automatically spot them (preliminary results are quite good).
> Once this is done, we can start asking interesting questions, such as:
> - What is the relative role of these formal experts to overall content
> contributed to Wikipedia?
> - Are formal experts' contributions "better"? (e.g. survive longer or
> result in increased quality score (per ORES)
> - Who are those formal experts? anonymous contributors? registered users?
> do they take additional roles within the community?
> - Formal experts' motivation
>
> Any other ideas for taking this research forward are more than welcome.
>
> Thank you,
> Ofer, Einat and Alex
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Aaron Halfaker-2
> result in increased quality score (per ORES)

<3  rock on

On Mon, Jul 10, 2017 at 3:15 PM, Shani Evenstein <[hidden email]>
wrote:

> Hi Alex,
>
> Welcome to the community!
>
> I am based at Tel Aviv University, where I teach 2 Wiki courses I developed
> and reseach Wikipedia & Wikidata (among others). If there's anything I can
> do to help, I'm a phone call away.  :-)
>
> Best,
> Shani.
>
> On 10 Jul 2017 23:02, "Alex Yarovoy" <[hidden email]> wrote:
>
> > Thank you Leila, Stuart, Pine
> > We will follow up on these comments and pointers
> >
> > A few additional words about this research -
> > Our narrow definition of formal expertise focuses on those with academic
> > qualifications who have published a scholarly work (i.e. appears in
> Google
> > Scholar) in the topic of the specific Wikipedia articles where one was
> > active.
> > We acknowledge that many experts do not have academic qualifications.
> > The choice of "formal" (i.e. academic in this context) expertise enabled
> a
> > concrete operationalization and measurement.
> > We welcome any ideas for pinpointing informal experts.
> >
> > We are currently in the first phase of research where we try to identify
> > these formal experts. We've spent considerable amount of time in
> > identifying 500 such experts, and now we use machine learning techniques
> to
> > automatically spot them (preliminary results are quite good).
> > Once this is done, we can start asking interesting questions, such as:
> > - What is the relative role of these formal experts to overall content
> > contributed to Wikipedia?
> > - Are formal experts' contributions "better"? (e.g. survive longer or
> > result in increased quality score (per ORES)
> > - Who are those formal experts? anonymous contributors? registered users?
> > do they take additional roles within the community?
> > - Formal experts' motivation
> >
> > Any other ideas for taking this research forward are more than welcome.
> >
> > Thank you,
> > Ofer, Einat and Alex
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Iolanda Pensa
In reply to this post by Alex Yarovoy
dear Ofer, Einat and Alex
to review articles and to write articles on Wikipedia
i would strongly suggest to focus on experts of that topic who are teachers (not necessarily people who have written a scientific article on that topic).

you are not looking for an advocate of a theory or for the last innovation, but rather you are looking for a person capable of providing an overview of a topic in a balanced way. a person who has written a scientific article about a specific topic will be very good at telling you what he/she thinks about that topic, but he/she will have troubles at providing the overview encyclopedic articles need. the teaching experience I think it is the best one to allow people to gain a real overview of a topic.

we noticed this issue within the project Wikipedia Primary School in which we involved around 30 experts in reviewing articles of Wikipedia  https://meta.wikimedia.org/wiki/Research:Wikipedia_Primary_School_SSAJRP_programme <https://meta.wikimedia.org/wiki/Research:Wikipedia_Primary_School_SSAJRP_programme>
but furthermore i discussed it with the director of Treccani (the Italian encyclopedia) and he explained me that the select for their articles not the best person who can present the topic (not the best - cutting edge/innovative - expert on the topic); furthermore they involve 7 other experts in the review process to make sure the article is balanced.

we also found very successful to have experts working outside wikipedia. we had them reviewing articles on a pdf to maintain their “status” (their status of experts works outside wikipedia, not inside since wikipedia has its own way of defining status) and to have them producing a review like they are used to without having them to learn how to use a wiki or how to interact with wikipedia (we provide them a pdf of the article, a template for the review, and a form to make their work under cc by-sa; we uploaded the review on Wikimedia commons and provide it on the discussion page of the article)

(here the review made during the project https://meta.wikimedia.org/wiki/Research:Wikipedia_Primary_School_SSAJRP_programme/Review <https://meta.wikimedia.org/wiki/Research:Wikipedia_Primary_School_SSAJRP_programme/Review>
and we are preparing now an article with Heather Ford about this research result)

all the best
iolanda/iopensa

> Il giorno 10 lug 2017, alle ore 22:02, Alex Yarovoy <[hidden email]> ha scritto:
>
> Thank you Leila, Stuart, Pine
> We will follow up on these comments and pointers
>
> A few additional words about this research -
> Our narrow definition of formal expertise focuses on those with academic
> qualifications who have published a scholarly work (i.e. appears in Google
> Scholar) in the topic of the specific Wikipedia articles where one was
> active.
> We acknowledge that many experts do not have academic qualifications.
> The choice of "formal" (i.e. academic in this context) expertise enabled a
> concrete operationalization and measurement.
> We welcome any ideas for pinpointing informal experts.
>
> We are currently in the first phase of research where we try to identify
> these formal experts. We've spent considerable amount of time in
> identifying 500 such experts, and now we use machine learning techniques to
> automatically spot them (preliminary results are quite good).
> Once this is done, we can start asking interesting questions, such as:
> - What is the relative role of these formal experts to overall content
> contributed to Wikipedia?
> - Are formal experts' contributions "better"? (e.g. survive longer or
> result in increased quality score (per ORES)
> - Who are those formal experts? anonymous contributors? registered users?
> do they take additional roles within the community?
> - Formal experts' motivation
>
> Any other ideas for taking this research forward are more than welcome.
>
> Thank you,
> Ofer, Einat and Alex
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Recognizing domain experts contribution to Wikipedia

Alex Yarovoy
In reply to this post by Alex Yarovoy
Thank you, iolanda, for highlighting the role of educators in contributing
to Wikipedia.
We do recognize that teachers often have highly relevant domain expertise.
Moreover, in many cases teachers are better than academics at providing an
overview of a topic in a balanced way.
Your approach for involving academic experts is interesting (i.e. having
them produce scientific reports that are linked - but not part of - the
wiki entry)
Thank you for sharing the links and report.We would be very interested in
reading your paper on the topic; please do share it when ready.

As for our own project -
As we wrote before, we take a very narrow definition of formal expertise
and focus on those publishing research papers on the particular topic of
the article they are contributing to.
Investigating the role of teachers in Wikipedia is a worthy quest, but
unfortunately it falls outside the scope of our current project.

Alex, Einat and Ofer
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l