gender balance of wikipedia citations

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

gender balance of wikipedia citations

Greg-2
Greetings!

I was looking for information about the gender balance of Wikipedia
citations and no one I've asked knows of any work on this topic. Do you?

I think this is an important question.

Here's what I've learned so far:

Wikipedia citations are currently in the form of text strings. There is
also an initiative to place citations in an annotated structured repository
(wikicite). I do not know the current status of wikicite or if/when this
could be used for this inquiry--either to examine all, or a sensible subset
of the citations.

My perspective is that understanding the gender balance is  necessary and
urgent. The balance could be better, the same, or worse than the citation
balances we already know, and the scale of the effect is quite large.

Is this a line of inquiry that the wikimedia/wikicite community is
interested in pursuing? If so, what is the best way to get started? Does
the WMF have the resources and interest to look into this matter inhouse?

Thanks for your thoughts.

Greg
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Kerry Raymond
Could you elaborate a bit more on what you mean by the gender balance of citations?

Are you talking about:

* proportion of male vs female authors of the source material used as citations in arbitrary articles>
*  the quality/quantity of citations in biography articles of men vs women?
* the quality/quantity of citations in articles that are gendered by some other criteria (e.g. reader interest, romantic comedy vs action film)?

Kerry

-----Original Message-----
From: Wiki-research-l [mailto:[hidden email]] On Behalf Of Greg
Sent: Thursday, 22 August 2019 1:19 PM
To: [hidden email]
Subject: [Wiki-research-l] gender balance of wikipedia citations

Greetings!

I was looking for information about the gender balance of Wikipedia citations and no one I've asked knows of any work on this topic. Do you?

I think this is an important question.

Here's what I've learned so far:

Wikipedia citations are currently in the form of text strings. There is also an initiative to place citations in an annotated structured repository (wikicite). I do not know the current status of wikicite or if/when this could be used for this inquiry--either to examine all, or a sensible subset of the citations.

My perspective is that understanding the gender balance is  necessary and urgent. The balance could be better, the same, or worse than the citation balances we already know, and the scale of the effect is quite large.

Is this a line of inquiry that the wikimedia/wikicite community is interested in pursuing? If so, what is the best way to get started? Does the WMF have the resources and interest to look into this matter inhouse?

Thanks for your thoughts.

Greg
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Greg-2
In reply to this post by Greg-2
Hi Kerry,
Those are all very interesting ways to look at this. I was thinking mostly
along the lines of your first bullet point, but I'd be interested in
research in any of those areas.

Thanks,
Greg

On Thu, Aug 22, 2019 at 5:00 AM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. gender balance of wikipedia citations (Greg)
>    2. Re: gender balance of wikipedia citations (Kerry Raymond)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 21 Aug 2019 20:19:18 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Greetings!
>
> I was looking for information about the gender balance of Wikipedia
> citations and no one I've asked knows of any work on this topic. Do you?
>
> I think this is an important question.
>
> Here's what I've learned so far:
>
> Wikipedia citations are currently in the form of text strings. There is
> also an initiative to place citations in an annotated structured repository
> (wikicite). I do not know the current status of wikicite or if/when this
> could be used for this inquiry--either to examine all, or a sensible subset
> of the citations.
>
> My perspective is that understanding the gender balance is  necessary and
> urgent. The balance could be better, the same, or worse than the citation
> balances we already know, and the scale of the effect is quite large.
>
> Is this a line of inquiry that the wikimedia/wikicite community is
> interested in pursuing? If so, what is the best way to get started? Does
> the WMF have the resources and interest to look into this matter inhouse?
>
> Thanks for your thoughts.
>
> Greg
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 22 Aug 2019 13:53:45 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> Content-Type: text/plain;       charset="UTF-8"
>
> Could you elaborate a bit more on what you mean by the gender balance of
> citations?
>
> Are you talking about:
>
> * proportion of male vs female authors of the source material used as
> citations in arbitrary articles>
> *  the quality/quantity of citations in biography articles of men vs women?
> * the quality/quantity of citations in articles that are gendered by some
> other criteria (e.g. reader interest, romantic comedy vs action film)?
>
> Kerry
>
> -----Original Message-----
> From: Wiki-research-l [mailto:[hidden email]]
> On Behalf Of Greg
> Sent: Thursday, 22 August 2019 1:19 PM
> To: [hidden email]
> Subject: [Wiki-research-l] gender balance of wikipedia citations
>
> Greetings!
>
> I was looking for information about the gender balance of Wikipedia
> citations and no one I've asked knows of any work on this topic. Do you?
>
> I think this is an important question.
>
> Here's what I've learned so far:
>
> Wikipedia citations are currently in the form of text strings. There is
> also an initiative to place citations in an annotated structured repository
> (wikicite). I do not know the current status of wikicite or if/when this
> could be used for this inquiry--either to examine all, or a sensible subset
> of the citations.
>
> My perspective is that understanding the gender balance is  necessary and
> urgent. The balance could be better, the same, or worse than the citation
> balances we already know, and the scale of the effect is quite large.
>
> Is this a line of inquiry that the wikimedia/wikicite community is
> interested in pursuing? If so, what is the best way to get started? Does
> the WMF have the resources and interest to look into this matter inhouse?
>
> Thanks for your thoughts.
>
> Greg
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 11
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Leila Zia
Hi Greg,

A few comments if you're going to go with "proportion of male vs
female authors of the source material used as citations in arbitrary
articles":

* Please differentiate between sex (female, male, ...) and gender
(woman, man, ...). My understanding from your initial email is that
you want to stay focused on gender, not sex.

* Unless you have reliable sources about the gender of an author, I
would not recommend trying to predict what the gender is. (As you may
know, this is not uncommon in social media studies, for example, to
predict the gender of the author based on their image or their name.
These approaches introduce biases and social challenges.)

* Re your question about whether WMF has resources to look into this
question in-house: I can't speak for the whole of WMF, however, I can
share more about the Research team's direction. As part of our future
work, we would like to "help contributors monitor violations of core
content policies and assess information reliability and bias both
granularly and at scale". [1] The question you proposed can fall under
assessing bias in content (considering citations as part of the
content). I expect us to focus first on the piece about violations of
core content policies and information reliability and come back to the
bias question later. As a result, we won't have bandwidth to do your
proposal in-house at the moment. Sorry about that.

I hope this helps.

Best,
Leila

[1] Section 2 of our Knowledge Integrity whitepaper:
https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf


On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:

>
> Hi Kerry,
> Those are all very interesting ways to look at this. I was thinking mostly
> along the lines of your first bullet point, but I'd be interested in
> research in any of those areas.
>
> Thanks,
> Greg
>
> On Thu, Aug 22, 2019 at 5:00 AM <[hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Greetings!
> >
> > I was looking for information about the gender balance of Wikipedia
> > citations and no one I've asked knows of any work on this topic. Do you?
> >
> > I think this is an important question.
> >
> > Here's what I've learned so far:
> >
> > Wikipedia citations are currently in the form of text strings. There is
> > also an initiative to place citations in an annotated structured repository
> > (wikicite). I do not know the current status of wikicite or if/when this
> > could be used for this inquiry--either to examine all, or a sensible subset
> > of the citations.
> >
> > My perspective is that understanding the gender balance is  necessary and
> > urgent. The balance could be better, the same, or worse than the citation
> > balances we already know, and the scale of the effect is quite large.
> >
> > Is this a line of inquiry that the wikimedia/wikicite community is
> > interested in pursuing? If so, what is the best way to get started? Does
> > the WMF have the resources and interest to look into this matter inhouse?
> >
> > Thanks for your thoughts.
> >
> > Greg
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > From: "Kerry Raymond" <[hidden email]>
> > To: "'Research into Wikimedia content and communities'"
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > Content-Type: text/plain;       charset="UTF-8"
> >
> > Could you elaborate a bit more on what you mean by the gender balance of
> > citations?
> >
> > Are you talking about:
> >
> > * proportion of male vs female authors of the source material used as
> > citations in arbitrary articles>
> > *  the quality/quantity of citations in biography articles of men vs women?
> > * the quality/quantity of citations in articles that are gendered by some
> > other criteria (e.g. reader interest, romantic comedy vs action film)?
> >
> > Kerry
> >
> > -----Original Message-----
> > From: Wiki-research-l [mailto:[hidden email]]
> > On Behalf Of Greg
> > Sent: Thursday, 22 August 2019 1:19 PM
> > To: [hidden email]
> > Subject: [Wiki-research-l] gender balance of wikipedia citations
> >
> > Greetings!
> >
> > I was looking for information about the gender balance of Wikipedia
> > citations and no one I've asked knows of any work on this topic. Do you?
> >
> > I think this is an important question.
> >
> > Here's what I've learned so far:
> >
> > Wikipedia citations are currently in the form of text strings. There is
> > also an initiative to place citations in an annotated structured repository
> > (wikicite). I do not know the current status of wikicite or if/when this
> > could be used for this inquiry--either to examine all, or a sensible subset
> > of the citations.
> >
> > My perspective is that understanding the gender balance is  necessary and
> > urgent. The balance could be better, the same, or worse than the citation
> > balances we already know, and the scale of the effect is quite large.
> >
> > Is this a line of inquiry that the wikimedia/wikicite community is
> > interested in pursuing? If so, what is the best way to get started? Does
> > the WMF have the resources and interest to look into this matter inhouse?
> >
> > Thanks for your thoughts.
> >
> > Greg
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 11
> > ************************************************
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Greg-2
In reply to this post by Greg-2
Hi Leila,

Thanks for your thoughts.

Having just read Troy Vettese's very powerful essay, Sexism in the Academy (
https://nplusonemag.com/issue-34/essays/sexism-in-the-academy/), I wish
this were a top priority.

I stumbled upon a study today--it came up in the Washington Post's
excellent series on gender bias in political science. The authors look at a
set of award winning political science books and the gender imbalance in
the citations drawn from google scholar.  I'm linking the piece here in
case anyone on this list is interested now, or in the future, in how the
patterns on Wikipedia compare.

Washington Post piece: "There’s a gender gap in who wins political science
book awards – and in how widely they’re cited"
https://www.washingtonpost.com/politics/2019/08/22/theres-gender-gap-who-wins-political-science-book-awards-how-widely-theyre-cited/
"Just as significantly, women’s award-winning books receive fewer scholarly
citations than men’s award-winning volumes — and this disparity has grown,
rather than shrunk, in recent years. Over the entire period, APSA
award-winning volumes by women averaged 43 percent fewer citations per year
than those by male authors."

Paper: "Winning awards and gaining recognition: An impact analysis of APSA
section book prizes"
https://www.sciencedirect.com/science/article/abs/pii/S0362331918300867


Best,
Greg

On Thu, Aug 22, 2019 at 3:44 PM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. Re: gender balance of wikipedia citations (Greg)
>    2. Re: gender balance of wikipedia citations (Leila Zia)
>    3. Wikimania 2019 disinformation meetup follow-up (Leila Zia)
>    4. Upcoming Research Newsletter (special issue on gender gap
>       research): New papers open for review (Mohammed Sadat Abdulai)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 22 Aug 2019 09:57:15 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <CAOO9DNuSYzzaVwcdqiWA7pj671z3N43XOSwv6DtW0SxWg=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Hi Kerry,
> Those are all very interesting ways to look at this. I was thinking mostly
> along the lines of your first bullet point, but I'd be interested in
> research in any of those areas.
>
> Thanks,
> Greg
>
> On Thu, Aug 22, 2019 at 5:00 AM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Greetings!
> >
> > I was looking for information about the gender balance of Wikipedia
> > citations and no one I've asked knows of any work on this topic. Do you?
> >
> > I think this is an important question.
> >
> > Here's what I've learned so far:
> >
> > Wikipedia citations are currently in the form of text strings. There is
> > also an initiative to place citations in an annotated structured
> repository
> > (wikicite). I do not know the current status of wikicite or if/when this
> > could be used for this inquiry--either to examine all, or a sensible
> subset
> > of the citations.
> >
> > My perspective is that understanding the gender balance is  necessary and
> > urgent. The balance could be better, the same, or worse than the citation
> > balances we already know, and the scale of the effect is quite large.
> >
> > Is this a line of inquiry that the wikimedia/wikicite community is
> > interested in pursuing? If so, what is the best way to get started? Does
> > the WMF have the resources and interest to look into this matter inhouse?
> >
> > Thanks for your thoughts.
> >
> > Greg
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > From: "Kerry Raymond" <[hidden email]>
> > To: "'Research into Wikimedia content and communities'"
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > Content-Type: text/plain;       charset="UTF-8"
> >
> > Could you elaborate a bit more on what you mean by the gender balance of
> > citations?
> >
> > Are you talking about:
> >
> > * proportion of male vs female authors of the source material used as
> > citations in arbitrary articles>
> > *  the quality/quantity of citations in biography articles of men vs
> women?
> > * the quality/quantity of citations in articles that are gendered by some
> > other criteria (e.g. reader interest, romantic comedy vs action film)?
> >
> > Kerry
> >
> > -----Original Message-----
> > From: Wiki-research-l [mailto:
> [hidden email]]
> > On Behalf Of Greg
> > Sent: Thursday, 22 August 2019 1:19 PM
> > To: [hidden email]
> > Subject: [Wiki-research-l] gender balance of wikipedia citations
> >
> > Greetings!
> >
> > I was looking for information about the gender balance of Wikipedia
> > citations and no one I've asked knows of any work on this topic. Do you?
> >
> > I think this is an important question.
> >
> > Here's what I've learned so far:
> >
> > Wikipedia citations are currently in the form of text strings. There is
> > also an initiative to place citations in an annotated structured
> repository
> > (wikicite). I do not know the current status of wikicite or if/when this
> > could be used for this inquiry--either to examine all, or a sensible
> subset
> > of the citations.
> >
> > My perspective is that understanding the gender balance is  necessary and
> > urgent. The balance could be better, the same, or worse than the citation
> > balances we already know, and the scale of the effect is quite large.
> >
> > Is this a line of inquiry that the wikimedia/wikicite community is
> > interested in pursuing? If so, what is the best way to get started? Does
> > the WMF have the resources and interest to look into this matter inhouse?
> >
> > Thanks for your thoughts.
> >
> > Greg
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 11
> > ************************************************
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 22 Aug 2019 10:43:51 -0700
> From: Leila Zia <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <CAK0Oe2uCo70_=ma2b=2d+fvr4GseEVxOP0sh=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Hi Greg,
>
> A few comments if you're going to go with "proportion of male vs
> female authors of the source material used as citations in arbitrary
> articles":
>
> * Please differentiate between sex (female, male, ...) and gender
> (woman, man, ...). My understanding from your initial email is that
> you want to stay focused on gender, not sex.
>
> * Unless you have reliable sources about the gender of an author, I
> would not recommend trying to predict what the gender is. (As you may
> know, this is not uncommon in social media studies, for example, to
> predict the gender of the author based on their image or their name.
> These approaches introduce biases and social challenges.)
>
> * Re your question about whether WMF has resources to look into this
> question in-house: I can't speak for the whole of WMF, however, I can
> share more about the Research team's direction. As part of our future
> work, we would like to "help contributors monitor violations of core
> content policies and assess information reliability and bias both
> granularly and at scale". [1] The question you proposed can fall under
> assessing bias in content (considering citations as part of the
> content). I expect us to focus first on the piece about violations of
> core content policies and information reliability and come back to the
> bias question later. As a result, we won't have bandwidth to do your
> proposal in-house at the moment. Sorry about that.
>
> I hope this helps.
>
> Best,
> Leila
>
> [1] Section 2 of our Knowledge Integrity whitepaper:
>
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
>
>
> On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> >
> > Hi Kerry,
> > Those are all very interesting ways to look at this. I was thinking
> mostly
> > along the lines of your first bullet point, but I'd be interested in
> > research in any of those areas.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 5:00 AM <
> [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of Wikipedia
> > > citations and no one I've asked knows of any work on this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings. There is
> > > also an initiative to place citations in an annotated structured
> repository
> > > (wikicite). I do not know the current status of wikicite or if/when
> this
> > > could be used for this inquiry--either to examine all, or a sensible
> subset
> > > of the citations.
> > >
> > > My perspective is that understanding the gender balance is  necessary
> and
> > > urgent. The balance could be better, the same, or worse than the
> citation
> > > balances we already know, and the scale of the effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> Does
> > > the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > From: "Kerry Raymond" <[hidden email]>
> > > To: "'Research into Wikimedia content and communities'"
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > Content-Type: text/plain;       charset="UTF-8"
> > >
> > > Could you elaborate a bit more on what you mean by the gender balance
> of
> > > citations?
> > >
> > > Are you talking about:
> > >
> > > * proportion of male vs female authors of the source material used as
> > > citations in arbitrary articles>
> > > *  the quality/quantity of citations in biography articles of men vs
> women?
> > > * the quality/quantity of citations in articles that are gendered by
> some
> > > other criteria (e.g. reader interest, romantic comedy vs action film)?
> > >
> > > Kerry
> > >
> > > -----Original Message-----
> > > From: Wiki-research-l [mailto:
> [hidden email]]
> > > On Behalf Of Greg
> > > Sent: Thursday, 22 August 2019 1:19 PM
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of Wikipedia
> > > citations and no one I've asked knows of any work on this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings. There is
> > > also an initiative to place citations in an annotated structured
> repository
> > > (wikicite). I do not know the current status of wikicite or if/when
> this
> > > could be used for this inquiry--either to examine all, or a sensible
> subset
> > > of the citations.
> > >
> > > My perspective is that understanding the gender balance is  necessary
> and
> > > urgent. The balance could be better, the same, or worse than the
> citation
> > > balances we already know, and the scale of the effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> Does
> > > the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > ************************************************
> > >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 22 Aug 2019 13:36:17 -0700
> From: Leila Zia <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: [Wiki-research-l] Wikimania 2019 disinformation meetup
>         follow-up
> Message-ID:
>         <CAK0Oe2sodYJpkuhSqgo3dtfDr=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Hi,
>
> This message is for those of you who attended the disinformation
> meet-up [0] in Wikimania 2019 [1] or others who may be interested.
>
> * The notes from our meet-up are now posted in the bottom of the page [0].
>
> * I was tasked to see if space.wmflabs.org is the place for us to
> continue conversations about this topic. The answer is yes. Thanks to
> the help of Elena Lappen, we now have a dedicated subcategory for
> disinformation:
> https://discuss-space.wmflabs.org/c/research/disinformation . Feel
> free to subscribe, watch, and/or post new topics if you're involved in
> this space.
>
> * If you are new to this conversation, please read the purpose of the
> subcategory at
> https://discuss-space.wmflabs.org/t/about-the-disinformation-category/949
> and welcome! :)
>
> Best,
> Leila
>
> [0] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation
> [1] https://wikimania.wikimedia.org/wiki/2019:Program
>
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 22 Aug 2019 22:43:53 +0000 (UTC)
> From: Mohammed Sadat Abdulai <[hidden email]>
> To: Research Into Wikimedia Content and Communities
>         <[hidden email]>
> Subject: [Wiki-research-l] Upcoming Research Newsletter (special issue
>         on gender gap research): New papers open for review
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=UTF-8
>
>  Hi everyone,
> We’re preparing for the August 2019 research newsletter and looking for
> contributors. Please take a look at
> https://etherpad.wikimedia.org/p/WRN201908 and add your name next to any
> paper you are interested in covering. Our target publication date is on 31
> August 11:59 UTC. As usual, short notes and one-paragraph reviews are most
> welcome.
>  For the August edition, we are planning a special issue focusing mainly
> on recent gender gap/gender bias research. (Upcoming special issues topics
> may include health and education.) There are about 20 papers from this area
> on our todo list which will all be covered in the August issue, either as a
> mere list item or - with your help - in form of a more informative writeup
> or review. They include:
>    - Analyzing Gender Stereotyping in Bollywood Movies
>
>    - Breaking the glass ceiling on Wikipedia| journal
>
>    - Breastfeeding, Authority, and Genre: Women's Ethos in Wikipedia and
> Blogs
>
>    - Cyberfeminism on Wikipedia: Visibility and deliberation in feminist
> Wikiprojects
>
>    - Gender and deletion on Wikipedia
>
>    - Gender imbalance and Wikipedia
>
>    - Gender Markers in Wikipedia Usernames
>
>    - How do students trust Wikipedia? An examination across genders
>
>    - Investigating the Gender Pronoun Gap in Wikipedia
>
>    - It’s Not What You Think: Gender Bias in Information about Fortune
> 1000 CEOs on Wikipedia
>
>    - Mapping and Bridging the Gender Gap: An Ethnographic Study of Indian
> Wikipedians and Their Motivations to Contribute
>
>    - People Who Can Take It: How Women Wikipedians Negotiate and Navigate
> Safety
>
>    - Redressing Gender Inequities on Wikipedia Through an Editathon
>
>    - Similar Gaps, Different Origins? Women Readers and Editors at Greek
> Wikipedia
>
>    - Simulation Experiments on (the Absence of) Ratings Bias in Reputation
> Systems
>
>    - The Gendered Presentation of Professions on Wikipedia
>
>    - Who Counts as a Notable Sociologist on Wikipedia? Gender, Race, and
> the “Professor Test”
>
>    - Who Wants to Read This?: A Method for Measuring Topical
> Representativeness in User Generated Content Systems
>
>    - Women and Wikipedia. Diversifying Editors and Enhancing Content
> through Library Edit-a-Thons
>
> Masssly and Tilman Bayer
>
> [1] Research:Newsletter - Meta[2] WikiResearch (@WikiResearch) on Twitter
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 12
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Kerry Raymond
In reply to this post by Leila Zia
Yes, that was my thought. It would be difficult to know the sex (or the gender) of an author name on a paper. There would inevitably be a lot that you could not determine. And certainly in the sciences multi-author pages are the norm and even where you did know the sex/gender of all, do you assign some part-score? E.g. 0 for all male, 1 for all female, 0.6 for 3 women and 2 men.

But I am curious why you are asking the question? That the writing/research of women is under-represented in Wikipedia citations? If so, without conducting any research, I'd say "yes it is under-represented". But my reason would be because women are under-represented as writers/researchers in the first place.  And certainly the older the source, the more likely it is to be written by a man. So to investigate gender bias in citations in Wikipedia, you would have to estimate the proportion of men/women (or at least their outputs) over time in a given discipline and then ask the question, "taking into account of the time of publication of a citation and the proportion of men/women active in this discipline at that time, do Wikipedia citations show a sex/gender basis?". Hmm ... very tricky.

I'd be inclined to suggest starting with a much simpler task. Pick a discipline (preferably one with a professional society who can tell your their estimate of current male/female ratio over (say) the past 5 years), limit the Wikipedia articles to topics in that discipline, and limit the citations to those published within the last 5 years. Indeed, perhaps limiting it to publications that are principally from the same country(s) as the professional society from which you get the data (as clearly men/women's participation in any discipline can vary with different countries for cultural reasons). Then you have some way to gauge whether Wikipedia is showing more or less gender bias in its citations than the discipline itself exhibits through publication. Quite a challenge!

And of course, it is not Wikipedia that adds citations. It is individual contributor who add citations. Does the sex/gender of the contributor have any correlation to any observed bias? Again, the task is made more difficult because a lot of Wikipedians don't identify their sex/gender.

The other thing to be alert to is the difference in how (I believe) Wikipedians cite compared to researchers. As a researcher, I will of course be reading papers in my field all the time and what I read will influence my subsequent work. Therefore when I write about my research, my citations are referring to papers that I have already read and whose authors may be familiar to me from their other work, having met them at a conferences, private correspondence, etc. However as a Wikipedian, I am only partially operating that way (mostly when I write new articles or significantly expand them, that is, when I am doing the research). A lot of the time I am adding citations relating to content other people (often new users) have added/changed without citation. These come up on my watchlist all the time. What do I do? Of course I could revert saying "no citation provided", but that's not the way to encourage new contributors nor to grow the encyclopedia, so if the information seems plausible (not obviously vandalism), I will attempt to find a citation for it (using tools like Google and other topic-specialise search tools). This is what I call "lucky dip" mode of citing as obviously I have no idea what the source was for the original contributor. The sources I find from my search may not already be known to me (frequently they are not). Or to summarise, IMHO, researchers (or Wikipedians in "new content mode") cite a source already known to them and whose authors may be known to them and could consciously or unconsciously engage in some discrimination in citation based on sex/gender or other criteria, whereas Wikipedians in "updating mode" are likely to be citing a source not previously known to them and may be happy just to have found a source and are unlikely to be spending a lot of their time researching the authors of that source to be extent they could then consciously or unconsciously exercise discrimination on sex/gender. If I invest any extra effort in such a situations, it's probably because the wording of the source is a close match to the Wikipedia article which begs the question of copyright violation (which needs to be dealt with by deletion or rewriting) or being a Wikipedia mirror (which is obviously not an acceptable citation).

So I suspect whether a citation was added by the same contributor as the content it supports or a subsequent contributor probably makes a difference to the likelihood of conscious/unconscious discrimination.

Also, finally, often Wikipedia cites web pages and other sources that do not have any individual authorship, e.g. government websites. Remember that Wikipedia prefers open citations over paywalled citations and a lot of the publications behind paywalls are individually authored.

Your proposed research has a lot of interesting challenges and a number of limitations. I'm not saying don't do it, but I am saying start very small and see if you can find any evidence to support your hypothesis before embarking on a larger study. Because contributor behaviour is what you are trying to study, you probably need to do both quantitative and qualitative experiments. E.g. I have described the two modes of citation I do, but I cannot say how typical my behaviour is.

Kerry

-----Original Message-----
From: Wiki-research-l [mailto:[hidden email]] On Behalf Of Leila Zia
Sent: Friday, 23 August 2019 3:44 AM
To: Research into Wikimedia content and communities <[hidden email]>
Subject: Re: [Wiki-research-l] gender balance of wikipedia citations

Hi Greg,

A few comments if you're going to go with "proportion of male vs female authors of the source material used as citations in arbitrary
articles":

* Please differentiate between sex (female, male, ...) and gender (woman, man, ...). My understanding from your initial email is that you want to stay focused on gender, not sex.

* Unless you have reliable sources about the gender of an author, I would not recommend trying to predict what the gender is. (As you may know, this is not uncommon in social media studies, for example, to predict the gender of the author based on their image or their name.
These approaches introduce biases and social challenges.)

* Re your question about whether WMF has resources to look into this question in-house: I can't speak for the whole of WMF, however, I can share more about the Research team's direction. As part of our future work, we would like to "help contributors monitor violations of core content policies and assess information reliability and bias both granularly and at scale". [1] The question you proposed can fall under assessing bias in content (considering citations as part of the content). I expect us to focus first on the piece about violations of core content policies and information reliability and come back to the bias question later. As a result, we won't have bandwidth to do your proposal in-house at the moment. Sorry about that.

I hope this helps.

Best,
Leila

[1] Section 2 of our Knowledge Integrity whitepaper:
https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf


On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:

>
> Hi Kerry,
> Those are all very interesting ways to look at this. I was thinking
> mostly along the lines of your first bullet point, but I'd be
> interested in research in any of those areas.
>
> Thanks,
> Greg
>
> On Thu, Aug 22, 2019 at 5:00 AM
> <[hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> >
> >
> > --------------------------------------------------------------------
> > --
> >
> > Message: 1
> > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Greetings!
> >
> > I was looking for information about the gender balance of Wikipedia
> > citations and no one I've asked knows of any work on this topic. Do you?
> >
> > I think this is an important question.
> >
> > Here's what I've learned so far:
> >
> > Wikipedia citations are currently in the form of text strings. There
> > is also an initiative to place citations in an annotated structured
> > repository (wikicite). I do not know the current status of wikicite
> > or if/when this could be used for this inquiry--either to examine
> > all, or a sensible subset of the citations.
> >
> > My perspective is that understanding the gender balance is  
> > necessary and urgent. The balance could be better, the same, or
> > worse than the citation balances we already know, and the scale of the effect is quite large.
> >
> > Is this a line of inquiry that the wikimedia/wikicite community is
> > interested in pursuing? If so, what is the best way to get started?
> > Does the WMF have the resources and interest to look into this matter inhouse?
> >
> > Thanks for your thoughts.
> >
> > Greg
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > From: "Kerry Raymond" <[hidden email]>
> > To: "'Research into Wikimedia content and communities'"
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > Content-Type: text/plain;       charset="UTF-8"
> >
> > Could you elaborate a bit more on what you mean by the gender
> > balance of citations?
> >
> > Are you talking about:
> >
> > * proportion of male vs female authors of the source material used
> > as citations in arbitrary articles>
> > *  the quality/quantity of citations in biography articles of men vs women?
> > * the quality/quantity of citations in articles that are gendered by
> > some other criteria (e.g. reader interest, romantic comedy vs action film)?
> >
> > Kerry
> >
> > -----Original Message-----
> > From: Wiki-research-l
> > [mailto:[hidden email]]
> > On Behalf Of Greg
> > Sent: Thursday, 22 August 2019 1:19 PM
> > To: [hidden email]
> > Subject: [Wiki-research-l] gender balance of wikipedia citations
> >
> > Greetings!
> >
> > I was looking for information about the gender balance of Wikipedia
> > citations and no one I've asked knows of any work on this topic. Do you?
> >
> > I think this is an important question.
> >
> > Here's what I've learned so far:
> >
> > Wikipedia citations are currently in the form of text strings. There
> > is also an initiative to place citations in an annotated structured
> > repository (wikicite). I do not know the current status of wikicite
> > or if/when this could be used for this inquiry--either to examine
> > all, or a sensible subset of the citations.
> >
> > My perspective is that understanding the gender balance is  
> > necessary and urgent. The balance could be better, the same, or
> > worse than the citation balances we already know, and the scale of the effect is quite large.
> >
> > Is this a line of inquiry that the wikimedia/wikicite community is
> > interested in pursuing? If so, what is the best way to get started?
> > Does the WMF have the resources and interest to look into this matter inhouse?
> >
> > Thanks for your thoughts.
> >
> > Greg
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 11
> > ************************************************
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Greg-2
In reply to this post by Greg-2
Wow, Kerry! Thank you for taking the time to write all these thoughts out.

I'm asking the question because I'm concerned that the gender balance of
the authors being cited on wikipedia is different from the already quite
bad patterns in academia. My fear is that the citation gender imbalance on
Wikipedia is more pronounced. If so, it is not just perpetuating the
problem, but making it worse by surfacing certain authors and ideas even
more frequently, or hardly at all. I would like to know if this is the
case, and if so, how big the effect is.

In my last message, I mention a study about a set of award-winning
political science books (the researchers study the citation gender
imbalance for that set). I just saw this study today, but I began to think
that it/the set of works--or some similar set of titles--could possibly be
a good place to begin, especially if the original researchers were willing
to share the list of titles/authors/gender/etc that they put
together/worked with. Then it seems it would mostly be a matter of figuring
out how to understand how those titles are cited on Wikipedia--through
either the citation dataset or wikicite--to see if/how the citation
patterns differ (i.e., if the works by women/men are cited more
frequently/at the same rate/less frequently on Wikipedia than what the
researchers found in the original study).

This seems like it would be easier to do than what you propose, but perhaps
the idea is not sound. Until very recently, I thought I could find the
answer in an existing paper! I honestly don't know the best way to get the
answer, but I would like to know the answer and think it's important to
look at.

All of the things you bring up--from the gender of the editor, to the type
of editing being done, to the issues around multiple authors/paywalls/year
of publication/field--complicate the inquiry, and in particular a larger
one. I agree with what you say about doing something small first to see
what's there.

Thanks again for all your thoughts.
Greg



On Thu, Aug 22, 2019 at 9:41 PM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. Re: gender balance of wikipedia citations (Greg)
>    2. Re: gender balance of wikipedia citations (Kerry Raymond)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 22 Aug 2019 18:47:48 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Hi Leila,
>
> Thanks for your thoughts.
>
> Having just read Troy Vettese's very powerful essay, Sexism in the Academy
> (
> https://nplusonemag.com/issue-34/essays/sexism-in-the-academy/), I wish
> this were a top priority.
>
> I stumbled upon a study today--it came up in the Washington Post's
> excellent series on gender bias in political science. The authors look at a
> set of award winning political science books and the gender imbalance in
> the citations drawn from google scholar.  I'm linking the piece here in
> case anyone on this list is interested now, or in the future, in how the
> patterns on Wikipedia compare.
>
> Washington Post piece: "There’s a gender gap in who wins political science
> book awards – and in how widely they’re cited"
>
> https://www.washingtonpost.com/politics/2019/08/22/theres-gender-gap-who-wins-political-science-book-awards-how-widely-theyre-cited/
> "Just as significantly, women’s award-winning books receive fewer scholarly
> citations than men’s award-winning volumes — and this disparity has grown,
> rather than shrunk, in recent years. Over the entire period, APSA
> award-winning volumes by women averaged 43 percent fewer citations per year
> than those by male authors."
>
> Paper: "Winning awards and gaining recognition: An impact analysis of APSA
> section book prizes"
> https://www.sciencedirect.com/science/article/abs/pii/S0362331918300867
>
>
> Best,
> Greg
>
> On Thu, Aug 22, 2019 at 3:44 PM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Leila Zia)
> >    3. Wikimania 2019 disinformation meetup follow-up (Leila Zia)
> >    4. Upcoming Research Newsletter (special issue on gender gap
> >       research): New papers open for review (Mohammed Sadat Abdulai)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 22 Aug 2019 09:57:15 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <CAOO9DNuSYzzaVwcdqiWA7pj671z3N43XOSwv6DtW0SxWg=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Kerry,
> > Those are all very interesting ways to look at this. I was thinking
> mostly
> > along the lines of your first bullet point, but I'd be interested in
> > research in any of those areas.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 5:00 AM <
> > [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of Wikipedia
> > > citations and no one I've asked knows of any work on this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings. There is
> > > also an initiative to place citations in an annotated structured
> > repository
> > > (wikicite). I do not know the current status of wikicite or if/when
> this
> > > could be used for this inquiry--either to examine all, or a sensible
> > subset
> > > of the citations.
> > >
> > > My perspective is that understanding the gender balance is  necessary
> and
> > > urgent. The balance could be better, the same, or worse than the
> citation
> > > balances we already know, and the scale of the effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> Does
> > > the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > From: "Kerry Raymond" <[hidden email]>
> > > To: "'Research into Wikimedia content and communities'"
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > Content-Type: text/plain;       charset="UTF-8"
> > >
> > > Could you elaborate a bit more on what you mean by the gender balance
> of
> > > citations?
> > >
> > > Are you talking about:
> > >
> > > * proportion of male vs female authors of the source material used as
> > > citations in arbitrary articles>
> > > *  the quality/quantity of citations in biography articles of men vs
> > women?
> > > * the quality/quantity of citations in articles that are gendered by
> some
> > > other criteria (e.g. reader interest, romantic comedy vs action film)?
> > >
> > > Kerry
> > >
> > > -----Original Message-----
> > > From: Wiki-research-l [mailto:
> > [hidden email]]
> > > On Behalf Of Greg
> > > Sent: Thursday, 22 August 2019 1:19 PM
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of Wikipedia
> > > citations and no one I've asked knows of any work on this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings. There is
> > > also an initiative to place citations in an annotated structured
> > repository
> > > (wikicite). I do not know the current status of wikicite or if/when
> this
> > > could be used for this inquiry--either to examine all, or a sensible
> > subset
> > > of the citations.
> > >
> > > My perspective is that understanding the gender balance is  necessary
> and
> > > urgent. The balance could be better, the same, or worse than the
> citation
> > > balances we already know, and the scale of the effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> Does
> > > the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > ************************************************
> > >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 22 Aug 2019 10:43:51 -0700
> > From: Leila Zia <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <CAK0Oe2uCo70_=ma2b=2d+fvr4GseEVxOP0sh=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Greg,
> >
> > A few comments if you're going to go with "proportion of male vs
> > female authors of the source material used as citations in arbitrary
> > articles":
> >
> > * Please differentiate between sex (female, male, ...) and gender
> > (woman, man, ...). My understanding from your initial email is that
> > you want to stay focused on gender, not sex.
> >
> > * Unless you have reliable sources about the gender of an author, I
> > would not recommend trying to predict what the gender is. (As you may
> > know, this is not uncommon in social media studies, for example, to
> > predict the gender of the author based on their image or their name.
> > These approaches introduce biases and social challenges.)
> >
> > * Re your question about whether WMF has resources to look into this
> > question in-house: I can't speak for the whole of WMF, however, I can
> > share more about the Research team's direction. As part of our future
> > work, we would like to "help contributors monitor violations of core
> > content policies and assess information reliability and bias both
> > granularly and at scale". [1] The question you proposed can fall under
> > assessing bias in content (considering citations as part of the
> > content). I expect us to focus first on the piece about violations of
> > core content policies and information reliability and come back to the
> > bias question later. As a result, we won't have bandwidth to do your
> > proposal in-house at the moment. Sorry about that.
> >
> > I hope this helps.
> >
> > Best,
> > Leila
> >
> > [1] Section 2 of our Knowledge Integrity whitepaper:
> >
> >
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
> >
> >
> > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > mostly
> > > along the lines of your first bullet point, but I'd be interested in
> > > research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM <
> > [hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more specific
> > > > than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > >
> ----------------------------------------------------------------------
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> is
> > > > also an initiative to place citations in an annotated structured
> > repository
> > > > (wikicite). I do not know the current status of wikicite or if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender balance
> > of
> > > > citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used as
> > > > citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men vs
> > women?
> > > > * the quality/quantity of citations in articles that are gendered by
> > some
> > > > other criteria (e.g. reader interest, romantic comedy vs action
> film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l [mailto:
> > [hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> is
> > > > also an initiative to place citations in an annotated structured
> > repository
> > > > (wikicite). I do not know the current status of wikicite or if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Thu, 22 Aug 2019 13:36:17 -0700
> > From: Leila Zia <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>
> > Subject: [Wiki-research-l] Wikimania 2019 disinformation meetup
> >         follow-up
> > Message-ID:
> >         <CAK0Oe2sodYJpkuhSqgo3dtfDr=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi,
> >
> > This message is for those of you who attended the disinformation
> > meet-up [0] in Wikimania 2019 [1] or others who may be interested.
> >
> > * The notes from our meet-up are now posted in the bottom of the page
> [0].
> >
> > * I was tasked to see if space.wmflabs.org is the place for us to
> > continue conversations about this topic. The answer is yes. Thanks to
> > the help of Elena Lappen, we now have a dedicated subcategory for
> > disinformation:
> > https://discuss-space.wmflabs.org/c/research/disinformation . Feel
> > free to subscribe, watch, and/or post new topics if you're involved in
> > this space.
> >
> > * If you are new to this conversation, please read the purpose of the
> > subcategory at
> >
> https://discuss-space.wmflabs.org/t/about-the-disinformation-category/949
> > and welcome! :)
> >
> > Best,
> > Leila
> >
> > [0] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation
> > [1] https://wikimania.wikimedia.org/wiki/2019:Program
> >
> >
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Thu, 22 Aug 2019 22:43:53 +0000 (UTC)
> > From: Mohammed Sadat Abdulai <[hidden email]>
> > To: Research Into Wikimedia Content and Communities
> >         <[hidden email]>
> > Subject: [Wiki-research-l] Upcoming Research Newsletter (special issue
> >         on gender gap research): New papers open for review
> > Message-ID: <[hidden email]>
> > Content-Type: text/plain; charset=UTF-8
> >
> >  Hi everyone,
> > We’re preparing for the August 2019 research newsletter and looking for
> > contributors. Please take a look at
> > https://etherpad.wikimedia.org/p/WRN201908 and add your name next to any
> > paper you are interested in covering. Our target publication date is on
> 31
> > August 11:59 UTC. As usual, short notes and one-paragraph reviews are
> most
> > welcome.
> >  For the August edition, we are planning a special issue focusing mainly
> > on recent gender gap/gender bias research. (Upcoming special issues
> topics
> > may include health and education.) There are about 20 papers from this
> area
> > on our todo list which will all be covered in the August issue, either
> as a
> > mere list item or - with your help - in form of a more informative
> writeup
> > or review. They include:
> >    - Analyzing Gender Stereotyping in Bollywood Movies
> >
> >    - Breaking the glass ceiling on Wikipedia| journal
> >
> >    - Breastfeeding, Authority, and Genre: Women's Ethos in Wikipedia and
> > Blogs
> >
> >    - Cyberfeminism on Wikipedia: Visibility and deliberation in feminist
> > Wikiprojects
> >
> >    - Gender and deletion on Wikipedia
> >
> >    - Gender imbalance and Wikipedia
> >
> >    - Gender Markers in Wikipedia Usernames
> >
> >    - How do students trust Wikipedia? An examination across genders
> >
> >    - Investigating the Gender Pronoun Gap in Wikipedia
> >
> >    - It’s Not What You Think: Gender Bias in Information about Fortune
> > 1000 CEOs on Wikipedia
> >
> >    - Mapping and Bridging the Gender Gap: An Ethnographic Study of Indian
> > Wikipedians and Their Motivations to Contribute
> >
> >    - People Who Can Take It: How Women Wikipedians Negotiate and Navigate
> > Safety
> >
> >    - Redressing Gender Inequities on Wikipedia Through an Editathon
> >
> >    - Similar Gaps, Different Origins? Women Readers and Editors at Greek
> > Wikipedia
> >
> >    - Simulation Experiments on (the Absence of) Ratings Bias in
> Reputation
> > Systems
> >
> >    - The Gendered Presentation of Professions on Wikipedia
> >
> >    - Who Counts as a Notable Sociologist on Wikipedia? Gender, Race, and
> > the “Professor Test”
> >
> >    - Who Wants to Read This?: A Method for Measuring Topical
> > Representativeness in User Generated Content Systems
> >
> >    - Women and Wikipedia. Diversifying Editors and Enhancing Content
> > through Library Edit-a-Thons
> >
> > Masssly and Tilman Bayer
> >
> > [1] Research:Newsletter - Meta[2] WikiResearch (@WikiResearch) on Twitter
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 12
> > ************************************************
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 23 Aug 2019 14:41:09 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID: <001001d5596c$fe22a100$fa67e300$@gmail.com>
> Content-Type: text/plain;       charset="utf-8"
>
> Yes, that was my thought. It would be difficult to know the sex (or the
> gender) of an author name on a paper. There would inevitably be a lot that
> you could not determine. And certainly in the sciences multi-author pages
> are the norm and even where you did know the sex/gender of all, do you
> assign some part-score? E.g. 0 for all male, 1 for all female, 0.6 for 3
> women and 2 men.
>
> But I am curious why you are asking the question? That the
> writing/research of women is under-represented in Wikipedia citations? If
> so, without conducting any research, I'd say "yes it is under-represented".
> But my reason would be because women are under-represented as
> writers/researchers in the first place.  And certainly the older the
> source, the more likely it is to be written by a man. So to investigate
> gender bias in citations in Wikipedia, you would have to estimate the
> proportion of men/women (or at least their outputs) over time in a given
> discipline and then ask the question, "taking into account of the time of
> publication of a citation and the proportion of men/women active in this
> discipline at that time, do Wikipedia citations show a sex/gender basis?".
> Hmm ... very tricky.
>
> I'd be inclined to suggest starting with a much simpler task. Pick a
> discipline (preferably one with a professional society who can tell your
> their estimate of current male/female ratio over (say) the past 5 years),
> limit the Wikipedia articles to topics in that discipline, and limit the
> citations to those published within the last 5 years. Indeed, perhaps
> limiting it to publications that are principally from the same country(s)
> as the professional society from which you get the data (as clearly
> men/women's participation in any discipline can vary with different
> countries for cultural reasons). Then you have some way to gauge whether
> Wikipedia is showing more or less gender bias in its citations than the
> discipline itself exhibits through publication. Quite a challenge!
>
> And of course, it is not Wikipedia that adds citations. It is individual
> contributor who add citations. Does the sex/gender of the contributor have
> any correlation to any observed bias? Again, the task is made more
> difficult because a lot of Wikipedians don't identify their sex/gender.
>
> The other thing to be alert to is the difference in how (I believe)
> Wikipedians cite compared to researchers. As a researcher, I will of course
> be reading papers in my field all the time and what I read will influence
> my subsequent work. Therefore when I write about my research, my citations
> are referring to papers that I have already read and whose authors may be
> familiar to me from their other work, having met them at a conferences,
> private correspondence, etc. However as a Wikipedian, I am only partially
> operating that way (mostly when I write new articles or significantly
> expand them, that is, when I am doing the research). A lot of the time I am
> adding citations relating to content other people (often new users) have
> added/changed without citation. These come up on my watchlist all the time.
> What do I do? Of course I could revert saying "no citation provided", but
> that's not the way to encourage new contributors nor to grow the
> encyclopedia, so if the information seems plausible (not obviously
> vandalism), I will attempt to find a citation for it (using tools like
> Google and other topic-specialise search tools). This is what I call "lucky
> dip" mode of citing as obviously I have no idea what the source was for the
> original contributor. The sources I find from my search may not already be
> known to me (frequently they are not). Or to summarise, IMHO, researchers
> (or Wikipedians in "new content mode") cite a source already known to them
> and whose authors may be known to them and could consciously or
> unconsciously engage in some discrimination in citation based on sex/gender
> or other criteria, whereas Wikipedians in "updating mode" are likely to be
> citing a source not previously known to them and may be happy just to have
> found a source and are unlikely to be spending a lot of their time
> researching the authors of that source to be extent they could then
> consciously or unconsciously exercise discrimination on sex/gender. If I
> invest any extra effort in such a situations, it's probably because the
> wording of the source is a close match to the Wikipedia article which begs
> the question of copyright violation (which needs to be dealt with by
> deletion or rewriting) or being a Wikipedia mirror (which is obviously not
> an acceptable citation).
>
> So I suspect whether a citation was added by the same contributor as the
> content it supports or a subsequent contributor probably makes a difference
> to the likelihood of conscious/unconscious discrimination.
>
> Also, finally, often Wikipedia cites web pages and other sources that do
> not have any individual authorship, e.g. government websites. Remember that
> Wikipedia prefers open citations over paywalled citations and a lot of the
> publications behind paywalls are individually authored.
>
> Your proposed research has a lot of interesting challenges and a number of
> limitations. I'm not saying don't do it, but I am saying start very small
> and see if you can find any evidence to support your hypothesis before
> embarking on a larger study. Because contributor behaviour is what you are
> trying to study, you probably need to do both quantitative and qualitative
> experiments. E.g. I have described the two modes of citation I do, but I
> cannot say how typical my behaviour is.
>
> Kerry
>
> -----Original Message-----
> From: Wiki-research-l [mailto:[hidden email]]
> On Behalf Of Leila Zia
> Sent: Friday, 23 August 2019 3:44 AM
> To: Research into Wikimedia content and communities <
> [hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
>
> Hi Greg,
>
> A few comments if you're going to go with "proportion of male vs female
> authors of the source material used as citations in arbitrary
> articles":
>
> * Please differentiate between sex (female, male, ...) and gender (woman,
> man, ...). My understanding from your initial email is that you want to
> stay focused on gender, not sex.
>
> * Unless you have reliable sources about the gender of an author, I would
> not recommend trying to predict what the gender is. (As you may know, this
> is not uncommon in social media studies, for example, to predict the gender
> of the author based on their image or their name.
> These approaches introduce biases and social challenges.)
>
> * Re your question about whether WMF has resources to look into this
> question in-house: I can't speak for the whole of WMF, however, I can share
> more about the Research team's direction. As part of our future work, we
> would like to "help contributors monitor violations of core content
> policies and assess information reliability and bias both granularly and at
> scale". [1] The question you proposed can fall under assessing bias in
> content (considering citations as part of the content). I expect us to
> focus first on the piece about violations of core content policies and
> information reliability and come back to the bias question later. As a
> result, we won't have bandwidth to do your proposal in-house at the moment.
> Sorry about that.
>
> I hope this helps.
>
> Best,
> Leila
>
> [1] Section 2 of our Knowledge Integrity whitepaper:
>
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
>
>
> On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> >
> > Hi Kerry,
> > Those are all very interesting ways to look at this. I was thinking
> > mostly along the lines of your first bullet point, but I'd be
> > interested in research in any of those areas.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 5:00 AM
> > <[hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > >
> > >
> > > --------------------------------------------------------------------
> > > --
> > >
> > > Message: 1
> > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of Wikipedia
> > > citations and no one I've asked knows of any work on this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings. There
> > > is also an initiative to place citations in an annotated structured
> > > repository (wikicite). I do not know the current status of wikicite
> > > or if/when this could be used for this inquiry--either to examine
> > > all, or a sensible subset of the citations.
> > >
> > > My perspective is that understanding the gender balance is
> > > necessary and urgent. The balance could be better, the same, or
> > > worse than the citation balances we already know, and the scale of the
> effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> > > Does the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > From: "Kerry Raymond" <[hidden email]>
> > > To: "'Research into Wikimedia content and communities'"
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > Content-Type: text/plain;       charset="UTF-8"
> > >
> > > Could you elaborate a bit more on what you mean by the gender
> > > balance of citations?
> > >
> > > Are you talking about:
> > >
> > > * proportion of male vs female authors of the source material used
> > > as citations in arbitrary articles>
> > > *  the quality/quantity of citations in biography articles of men vs
> women?
> > > * the quality/quantity of citations in articles that are gendered by
> > > some other criteria (e.g. reader interest, romantic comedy vs action
> film)?
> > >
> > > Kerry
> > >
> > > -----Original Message-----
> > > From: Wiki-research-l
> > > [mailto:[hidden email]]
> > > On Behalf Of Greg
> > > Sent: Thursday, 22 August 2019 1:19 PM
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of Wikipedia
> > > citations and no one I've asked knows of any work on this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings. There
> > > is also an initiative to place citations in an annotated structured
> > > repository (wikicite). I do not know the current status of wikicite
> > > or if/when this could be used for this inquiry--either to examine
> > > all, or a sensible subset of the citations.
> > >
> > > My perspective is that understanding the gender balance is
> > > necessary and urgent. The balance could be better, the same, or
> > > worse than the citation balances we already know, and the scale of the
> effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> > > Does the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > ************************************************
> > >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 13
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Greg-2
In reply to this post by Greg-2
Hi all-

One more thing: on twitter, I was advised that this list and the wikicite
google group were the best places to discuss research around citations.
Although I would like to post about this line of inquiry to the wikicite
group, it appears to be a private group. As an outsider, I have not been
able to access/view the group content/or even see who owns the group and is
the correct person to contact. I have mentioned that I can not access the
group to the wikicite twitter handle, and received a 'like' (?) but no
response, and nothing has changed.

If you are a member of both lists, would you be willing to point to this
thread from the wikicite group? If the group is not open to the public, at
least the ideas will be there.

Many thanks,

Greg

On Fri, Aug 23, 2019 at 12:01 AM <
[hidden email]> wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. sockpuppets and how to find them sooner (Kerry Raymond)
>    2. Re: sockpuppets and how to find them sooner (Kerry Raymond)
>    3. Re: sockpuppets and how to find them sooner (RhinosF1)
>    4. Re: gender balance of wikipedia citations (Greg)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 23 Aug 2019 15:57:15 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID: <001b01d55977$9f51dbe0$ddf593a0$@gmail.com>
> Content-Type: text/plain;       charset="us-ascii"
>
> Currently, to open a sockpuppet investigation, you must name the two (or
> more) accounts that you believe to be sockpuppets with "clear, behavioural
> evidence of sock puppetry" which is typically in the form of pairs of edits
> that demonstrate similar edit behaviours that are unlikely to naturally
> occur. Now if you spend enough time on-wiki, you develop an intuition about
> behaviours you see on your watchlist and in article edit histories. Often I
> am highly suspicious that an account is a sockpuppet, but I cannot report
> them because I don't know which other account is involved.
>
>
>
> As a example, I recently encounted User:Shelati an account about 1 day old
> at that time with nearly 100 edits in that day all about 1-2 minutes apart,
> mostly making a similar change to a large number of Australian place
> infoboxes.
>
>
>
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati
> <
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&of
> fset=20190728053057&limit=100&target=Shelati
> <https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&offset=20190728053057&limit=100&target=Shelati>
> >
> &offset=20190728053057&limit=100&target=Shelati
>
>
>
> Genuine new users do not edit that quickly, do not use templates and do not
> mess structurally with infoboxes (at most they try to change the values).
> It
> "smelled" like a sockpuppet. However, as I did not recognise that pattern
> of
> edit behaviour as being that of any other user I was familiar with, it
> wasn't something I could report for sockpuppet investigation. Anyhow after
> about 2 weeks, the user was blocked as a sockpuppet. Someone must have
> noticed and figured out the other account:
>
>
>
>
> https://en.wikipedia.org/wiki/Wikipedia:Sockpuppet_investigations/Meganesia/
> Archive
>
>
>
> Two weeks and 1,279 edits later . that's over 1000 possibly problematic
> edits after I first suspected them. But that's nothing compared with
> another
> ongoing situation in which a very large number of different IPs are engaged
> in a pattern of problem edits on mostly Australian articles (a few
> different
> types of edits but an obvious "quack like a duck" situation). The IP number
> changes frequently (and one assumes deliberately). The edits potentially go
> back to 2013 but appear to have intensified in 2018/2019. Here's one user's
> summary of all the IP addresses involved, and the extent to which they have
> been cleaned up, given many thousands of edits are involved, see:
>
>
>
> https://en.wikipedia.org/wiki/User:IamNotU/History_cleanup
>
>
>
> As well as the damage done to the content (which harms the readers), these
> IP sockpuppets are consuming enormous amounts of effort to track them down
> and revert them, which could be more productively used to improve the
> content. We need better tools to foil these pests. So I want to put that
> challenge out to this list.
>
>
>
> Kerry
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 23 Aug 2019 16:26:43 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID: <002001d5597b$bd1f6b20$375e4160$@gmail.com>
> Content-Type: text/plain;       charset="us-ascii"
>
> To reply to my own question .
>
>
>
> Can we find a way to create a "signature" of an account's pattern of
> editing? Perhaps it might be a set of signatures, maybe one for the
> categories that the account appears to be active in, another for the type
> of
> edit, etc. Then if these signatures were calculated for all banned accounts
> or currently blocked accounts (or at least ones with a long enough
> contribution history to make it worthwhile - we're not interested in
> one-edit vandals), then we could have a tool that could be run to quickly
> compare one account against the signatures of banned/blocked accounts as
> well as the cumulative edits of a set of known sockpuppets (i.e. treat them
> as a single account) to determine if this may be a sockpuppet case meriting
> further investigation. I imagine that it would be too expensive
> computationally to actually run comparisons of the contribution histories
> of
> all "bad guy" accounts against the suspicious account which is why I
> propose
> a "signature" approach (but I'm happy to be told otherwise).
>
>
>
> If we had such a tool and it proves reasonably reliable in identifying
> likely sockpuppets (not asking for guarantees but close enough not to be a
> waste of time to investigate), then we could routinely use it on new
> accounts or reactivating accounts (i.e. possible sleeper accounts) once
> they
> have a long enough editing history to enable the tool to operate
> effectively
> to provide automated early warning of new/reactivating accounts appearing
> suspiciously similar to "bad guy" accounts.
>
>
>
> But this is a hard problem, both technically and socially (Assume Good
> Faith, Privacy, etc), so I welcome the thoughts of others.
>
>
>
> Kerry
>
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 23 Aug 2019 07:43:29 +0100
> From: RhinosF1 <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>, [hidden email]
> Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID:
>         <CAK3HvEMj8VCPu+uVUmPW+RepJ6E-OULaLPA15ALv9C=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Just a note that you can still go through warnings for vandalism etc. and
> report to AIV.
>
> Or at that edit speed, you may have a chance at AN at reporting for
> bot-like edits which will draw attention to the account.
>
> If you ever need help, things like #wikipedia-en-help on Freenode IRC exist
> so you can ask other users.
>
> RhinosF1
> Miraheze Volunteer
>
> On Fri, 23 Aug 2019 at 06:57, Kerry Raymond <[hidden email]>
> wrote:
>
> > Currently, to open a sockpuppet investigation, you must name the two (or
> > more) accounts that you believe to be sockpuppets with "clear,
> behavioural
> > evidence of sock puppetry" which is typically in the form of pairs of
> edits
> > that demonstrate similar edit behaviours that are unlikely to naturally
> > occur. Now if you spend enough time on-wiki, you develop an intuition
> about
> > behaviours you see on your watchlist and in article edit histories.
> Often I
> > am highly suspicious that an account is a sockpuppet, but I cannot report
> > them because I don't know which other account is involved.
> >
> >
> >
> > As a example, I recently encounted User:Shelati an account about 1 day
> old
> > at that time with nearly 100 edits in that day all about 1-2 minutes
> apart,
> > mostly making a similar change to a large number of Australian place
> > infoboxes.
> >
> >
> >
> > https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati
> > <
> >
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&of
> > fset=20190728053057&limit=100&target=Shelati
> > <
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&offset=20190728053057&limit=100&target=Shelati
> >
> > >
> > &offset=20190728053057&limit=100&target=Shelati
> >
> >
> >
> > Genuine new users do not edit that quickly, do not use templates and do
> not
> > mess structurally with infoboxes (at most they try to change the values).
> > It
> > "smelled" like a sockpuppet. However, as I did not recognise that pattern
> > of
> > edit behaviour as being that of any other user I was familiar with, it
> > wasn't something I could report for sockpuppet investigation. Anyhow
> after
> > about 2 weeks, the user was blocked as a sockpuppet. Someone must have
> > noticed and figured out the other account:
> >
> >
> >
> >
> >
> https://en.wikipedia.org/wiki/Wikipedia:Sockpuppet_investigations/Meganesia/
> > Archive
> >
> >
> >
> > Two weeks and 1,279 edits later . that's over 1000 possibly problematic
> > edits after I first suspected them. But that's nothing compared with
> > another
> > ongoing situation in which a very large number of different IPs are
> engaged
> > in a pattern of problem edits on mostly Australian articles (a few
> > different
> > types of edits but an obvious "quack like a duck" situation). The IP
> number
> > changes frequently (and one assumes deliberately). The edits potentially
> go
> > back to 2013 but appear to have intensified in 2018/2019. Here's one
> user's
> > summary of all the IP addresses involved, and the extent to which they
> have
> > been cleaned up, given many thousands of edits are involved, see:
> >
> >
> >
> > https://en.wikipedia.org/wiki/User:IamNotU/History_cleanup
> >
> >
> >
> > As well as the damage done to the content (which harms the readers),
> these
> > IP sockpuppets are consuming enormous amounts of effort to track them
> down
> > and revert them, which could be more productively used to improve the
> > content. We need better tools to foil these pests. So I want to put that
> > challenge out to this list.
> >
> >
> >
> > Kerry
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> --
> RhinosF1
> Miraheze Volunteer
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 23 Aug 2019 00:01:15 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Wow, Kerry! Thank you for taking the time to write all these thoughts out.
>
> I'm asking the question because I'm concerned that the gender balance of
> the authors being cited on wikipedia is different from the already quite
> bad patterns in academia. My fear is that the citation gender imbalance on
> Wikipedia is more pronounced. If so, it is not just perpetuating the
> problem, but making it worse by surfacing certain authors and ideas even
> more frequently, or hardly at all. I would like to know if this is the
> case, and if so, how big the effect is.
>
> In my last message, I mention a study about a set of award-winning
> political science books (the researchers study the citation gender
> imbalance for that set). I just saw this study today, but I began to think
> that it/the set of works--or some similar set of titles--could possibly be
> a good place to begin, especially if the original researchers were willing
> to share the list of titles/authors/gender/etc that they put
> together/worked with. Then it seems it would mostly be a matter of figuring
> out how to understand how those titles are cited on Wikipedia--through
> either the citation dataset or wikicite--to see if/how the citation
> patterns differ (i.e., if the works by women/men are cited more
> frequently/at the same rate/less frequently on Wikipedia than what the
> researchers found in the original study).
>
> This seems like it would be easier to do than what you propose, but perhaps
> the idea is not sound. Until very recently, I thought I could find the
> answer in an existing paper! I honestly don't know the best way to get the
> answer, but I would like to know the answer and think it's important to
> look at.
>
> All of the things you bring up--from the gender of the editor, to the type
> of editing being done, to the issues around multiple authors/paywalls/year
> of publication/field--complicate the inquiry, and in particular a larger
> one. I agree with what you say about doing something small first to see
> what's there.
>
> Thanks again for all your thoughts.
> Greg
>
>
>
> On Thu, Aug 22, 2019 at 9:41 PM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 22 Aug 2019 18:47:48 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Leila,
> >
> > Thanks for your thoughts.
> >
> > Having just read Troy Vettese's very powerful essay, Sexism in the
> Academy
> > (
> > https://nplusonemag.com/issue-34/essays/sexism-in-the-academy/), I wish
> > this were a top priority.
> >
> > I stumbled upon a study today--it came up in the Washington Post's
> > excellent series on gender bias in political science. The authors look
> at a
> > set of award winning political science books and the gender imbalance in
> > the citations drawn from google scholar.  I'm linking the piece here in
> > case anyone on this list is interested now, or in the future, in how the
> > patterns on Wikipedia compare.
> >
> > Washington Post piece: "There’s a gender gap in who wins political
> science
> > book awards – and in how widely they’re cited"
> >
> >
> https://www.washingtonpost.com/politics/2019/08/22/theres-gender-gap-who-wins-political-science-book-awards-how-widely-theyre-cited/
> > "Just as significantly, women’s award-winning books receive fewer
> scholarly
> > citations than men’s award-winning volumes — and this disparity has
> grown,
> > rather than shrunk, in recent years. Over the entire period, APSA
> > award-winning volumes by women averaged 43 percent fewer citations per
> year
> > than those by male authors."
> >
> > Paper: "Winning awards and gaining recognition: An impact analysis of
> APSA
> > section book prizes"
> > https://www.sciencedirect.com/science/article/abs/pii/S0362331918300867
> >
> >
> > Best,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 3:44 PM <
> > [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. Re: gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Leila Zia)
> > >    3. Wikimania 2019 disinformation meetup follow-up (Leila Zia)
> > >    4. Upcoming Research Newsletter (special issue on gender gap
> > >       research): New papers open for review (Mohammed Sadat Abdulai)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Thu, 22 Aug 2019 09:57:15 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <CAOO9DNuSYzzaVwcdqiWA7pj671z3N43XOSwv6DtW0SxWg=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > mostly
> > > along the lines of your first bullet point, but I'd be interested in
> > > research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more specific
> > > > than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > >
> ----------------------------------------------------------------------
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> is
> > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > (wikicite). I do not know the current status of wikicite or if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender balance
> > of
> > > > citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used as
> > > > citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men vs
> > > women?
> > > > * the quality/quantity of citations in articles that are gendered by
> > some
> > > > other criteria (e.g. reader interest, romantic comedy vs action
> film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l [mailto:
> > > [hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> is
> > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > (wikicite). I do not know the current status of wikicite or if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 10:43:51 -0700
> > > From: Leila Zia <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <CAK0Oe2uCo70_=ma2b=2d+fvr4GseEVxOP0sh=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Greg,
> > >
> > > A few comments if you're going to go with "proportion of male vs
> > > female authors of the source material used as citations in arbitrary
> > > articles":
> > >
> > > * Please differentiate between sex (female, male, ...) and gender
> > > (woman, man, ...). My understanding from your initial email is that
> > > you want to stay focused on gender, not sex.
> > >
> > > * Unless you have reliable sources about the gender of an author, I
> > > would not recommend trying to predict what the gender is. (As you may
> > > know, this is not uncommon in social media studies, for example, to
> > > predict the gender of the author based on their image or their name.
> > > These approaches introduce biases and social challenges.)
> > >
> > > * Re your question about whether WMF has resources to look into this
> > > question in-house: I can't speak for the whole of WMF, however, I can
> > > share more about the Research team's direction. As part of our future
> > > work, we would like to "help contributors monitor violations of core
> > > content policies and assess information reliability and bias both
> > > granularly and at scale". [1] The question you proposed can fall under
> > > assessing bias in content (considering citations as part of the
> > > content). I expect us to focus first on the piece about violations of
> > > core content policies and information reliability and come back to the
> > > bias question later. As a result, we won't have bandwidth to do your
> > > proposal in-house at the moment. Sorry about that.
> > >
> > > I hope this helps.
> > >
> > > Best,
> > > Leila
> > >
> > > [1] Section 2 of our Knowledge Integrity whitepaper:
> > >
> > >
> >
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
> > >
> > >
> > > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]>
> wrote:
> > > >
> > > > Hi Kerry,
> > > > Those are all very interesting ways to look at this. I was thinking
> > > mostly
> > > > along the lines of your first bullet point, but I'd be interested in
> > > > research in any of those areas.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Thu, Aug 22, 2019 at 5:00 AM <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Send Wiki-research-l mailing list submissions to
> > > > >         [hidden email]
> > > > >
> > > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > > >
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > > or, via email, send a message with subject or body 'help' to
> > > > >         [hidden email]
> > > > >
> > > > > You can reach the person managing the list at
> > > > >         [hidden email]
> > > > >
> > > > > When replying, please edit your Subject line so it is more specific
> > > > > than "Re: Contents of Wiki-research-l digest..."
> > > > >
> > > > >
> > > > > Today's Topics:
> > > > >
> > > > >    1. gender balance of wikipedia citations (Greg)
> > > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > > >
> > > > >
> > > > >
> > ----------------------------------------------------------------------
> > > > >
> > > > > Message: 1
> > > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > > From: Greg <[hidden email]>
> > > > > To: [hidden email]
> > > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > > Message-ID:
> > > > >         <
> > > > > [hidden email]
> >
> > > > > Content-Type: text/plain; charset="UTF-8"
> > > > >
> > > > > Greetings!
> > > > >
> > > > > I was looking for information about the gender balance of Wikipedia
> > > > > citations and no one I've asked knows of any work on this topic. Do
> > > you?
> > > > >
> > > > > I think this is an important question.
> > > > >
> > > > > Here's what I've learned so far:
> > > > >
> > > > > Wikipedia citations are currently in the form of text strings.
> There
> > is
> > > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > > (wikicite). I do not know the current status of wikicite or if/when
> > > this
> > > > > could be used for this inquiry--either to examine all, or a
> sensible
> > > subset
> > > > > of the citations.
> > > > >
> > > > > My perspective is that understanding the gender balance is
> necessary
> > > and
> > > > > urgent. The balance could be better, the same, or worse than the
> > > citation
> > > > > balances we already know, and the scale of the effect is quite
> large.
> > > > >
> > > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > > interested in pursuing? If so, what is the best way to get started?
> > > Does
> > > > > the WMF have the resources and interest to look into this matter
> > > inhouse?
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > Greg
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > Message: 2
> > > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > > From: "Kerry Raymond" <[hidden email]>
> > > > > To: "'Research into Wikimedia content and communities'"
> > > > >         <[hidden email]>
> > > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> citations
> > > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > > Content-Type: text/plain;       charset="UTF-8"
> > > > >
> > > > > Could you elaborate a bit more on what you mean by the gender
> balance
> > > of
> > > > > citations?
> > > > >
> > > > > Are you talking about:
> > > > >
> > > > > * proportion of male vs female authors of the source material used
> as
> > > > > citations in arbitrary articles>
> > > > > *  the quality/quantity of citations in biography articles of men
> vs
> > > women?
> > > > > * the quality/quantity of citations in articles that are gendered
> by
> > > some
> > > > > other criteria (e.g. reader interest, romantic comedy vs action
> > film)?
> > > > >
> > > > > Kerry
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wiki-research-l [mailto:
> > > [hidden email]]
> > > > > On Behalf Of Greg
> > > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > > To: [hidden email]
> > > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > >
> > > > > Greetings!
> > > > >
> > > > > I was looking for information about the gender balance of Wikipedia
> > > > > citations and no one I've asked knows of any work on this topic. Do
> > > you?
> > > > >
> > > > > I think this is an important question.
> > > > >
> > > > > Here's what I've learned so far:
> > > > >
> > > > > Wikipedia citations are currently in the form of text strings.
> There
> > is
> > > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > > (wikicite). I do not know the current status of wikicite or if/when
> > > this
> > > > > could be used for this inquiry--either to examine all, or a
> sensible
> > > subset
> > > > > of the citations.
> > > > >
> > > > > My perspective is that understanding the gender balance is
> necessary
> > > and
> > > > > urgent. The balance could be better, the same, or worse than the
> > > citation
> > > > > balances we already know, and the scale of the effect is quite
> large.
> > > > >
> > > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > > interested in pursuing? If so, what is the best way to get started?
> > > Does
> > > > > the WMF have the resources and interest to look into this matter
> > > inhouse?
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > Greg
> > > > > _______________________________________________
> > > > > Wiki-research-l mailing list
> > > > > [hidden email]
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > Subject: Digest Footer
> > > > >
> > > > > _______________________________________________
> > > > > Wiki-research-l mailing list
> > > > > [hidden email]
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > > ************************************************
> > > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 3
> > > Date: Thu, 22 Aug 2019 13:36:17 -0700
> > > From: Leila Zia <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: [Wiki-research-l] Wikimania 2019 disinformation meetup
> > >         follow-up
> > > Message-ID:
> > >         <CAK0Oe2sodYJpkuhSqgo3dtfDr=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi,
> > >
> > > This message is for those of you who attended the disinformation
> > > meet-up [0] in Wikimania 2019 [1] or others who may be interested.
> > >
> > > * The notes from our meet-up are now posted in the bottom of the page
> > [0].
> > >
> > > * I was tasked to see if space.wmflabs.org is the place for us to
> > > continue conversations about this topic. The answer is yes. Thanks to
> > > the help of Elena Lappen, we now have a dedicated subcategory for
> > > disinformation:
> > > https://discuss-space.wmflabs.org/c/research/disinformation . Feel
> > > free to subscribe, watch, and/or post new topics if you're involved in
> > > this space.
> > >
> > > * If you are new to this conversation, please read the purpose of the
> > > subcategory at
> > >
> >
> https://discuss-space.wmflabs.org/t/about-the-disinformation-category/949
> > > and welcome! :)
> > >
> > > Best,
> > > Leila
> > >
> > > [0] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation
> > > [1] https://wikimania.wikimedia.org/wiki/2019:Program
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 4
> > > Date: Thu, 22 Aug 2019 22:43:53 +0000 (UTC)
> > > From: Mohammed Sadat Abdulai <[hidden email]>
> > > To: Research Into Wikimedia Content and Communities
> > >         <[hidden email]>
> > > Subject: [Wiki-research-l] Upcoming Research Newsletter (special issue
> > >         on gender gap research): New papers open for review
> > > Message-ID: <[hidden email]>
> > > Content-Type: text/plain; charset=UTF-8
> > >
> > >  Hi everyone,
> > > We’re preparing for the August 2019 research newsletter and looking for
> > > contributors. Please take a look at
> > > https://etherpad.wikimedia.org/p/WRN201908 and add your name next to
> any
> > > paper you are interested in covering. Our target publication date is on
> > 31
> > > August 11:59 UTC. As usual, short notes and one-paragraph reviews are
> > most
> > > welcome.
> > >  For the August edition, we are planning a special issue focusing
> mainly
> > > on recent gender gap/gender bias research. (Upcoming special issues
> > topics
> > > may include health and education.) There are about 20 papers from this
> > area
> > > on our todo list which will all be covered in the August issue, either
> > as a
> > > mere list item or - with your help - in form of a more informative
> > writeup
> > > or review. They include:
> > >    - Analyzing Gender Stereotyping in Bollywood Movies
> > >
> > >    - Breaking the glass ceiling on Wikipedia| journal
> > >
> > >    - Breastfeeding, Authority, and Genre: Women's Ethos in Wikipedia
> and
> > > Blogs
> > >
> > >    - Cyberfeminism on Wikipedia: Visibility and deliberation in
> feminist
> > > Wikiprojects
> > >
> > >    - Gender and deletion on Wikipedia
> > >
> > >    - Gender imbalance and Wikipedia
> > >
> > >    - Gender Markers in Wikipedia Usernames
> > >
> > >    - How do students trust Wikipedia? An examination across genders
> > >
> > >    - Investigating the Gender Pronoun Gap in Wikipedia
> > >
> > >    - It’s Not What You Think: Gender Bias in Information about Fortune
> > > 1000 CEOs on Wikipedia
> > >
> > >    - Mapping and Bridging the Gender Gap: An Ethnographic Study of
> Indian
> > > Wikipedians and Their Motivations to Contribute
> > >
> > >    - People Who Can Take It: How Women Wikipedians Negotiate and
> Navigate
> > > Safety
> > >
> > >    - Redressing Gender Inequities on Wikipedia Through an Editathon
> > >
> > >    - Similar Gaps, Different Origins? Women Readers and Editors at
> Greek
> > > Wikipedia
> > >
> > >    - Simulation Experiments on (the Absence of) Ratings Bias in
> > Reputation
> > > Systems
> > >
> > >    - The Gendered Presentation of Professions on Wikipedia
> > >
> > >    - Who Counts as a Notable Sociologist on Wikipedia? Gender, Race,
> and
> > > the “Professor Test”
> > >
> > >    - Who Wants to Read This?: A Method for Measuring Topical
> > > Representativeness in User Generated Content Systems
> > >
> > >    - Women and Wikipedia. Diversifying Editors and Enhancing Content
> > > through Library Edit-a-Thons
> > >
> > > Masssly and Tilman Bayer
> > >
> > > [1] Research:Newsletter - Meta[2] WikiResearch (@WikiResearch) on
> Twitter
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 12
> > > ************************************************
> > >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Fri, 23 Aug 2019 14:41:09 +1000
> > From: "Kerry Raymond" <[hidden email]>
> > To: "'Research into Wikimedia content and communities'"
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <001001d5596c$fe22a100$fa67e300$@gmail.com>
> > Content-Type: text/plain;       charset="utf-8"
> >
> > Yes, that was my thought. It would be difficult to know the sex (or the
> > gender) of an author name on a paper. There would inevitably be a lot
> that
> > you could not determine. And certainly in the sciences multi-author pages
> > are the norm and even where you did know the sex/gender of all, do you
> > assign some part-score? E.g. 0 for all male, 1 for all female, 0.6 for 3
> > women and 2 men.
> >
> > But I am curious why you are asking the question? That the
> > writing/research of women is under-represented in Wikipedia citations? If
> > so, without conducting any research, I'd say "yes it is
> under-represented".
> > But my reason would be because women are under-represented as
> > writers/researchers in the first place.  And certainly the older the
> > source, the more likely it is to be written by a man. So to investigate
> > gender bias in citations in Wikipedia, you would have to estimate the
> > proportion of men/women (or at least their outputs) over time in a given
> > discipline and then ask the question, "taking into account of the time of
> > publication of a citation and the proportion of men/women active in this
> > discipline at that time, do Wikipedia citations show a sex/gender
> basis?".
> > Hmm ... very tricky.
> >
> > I'd be inclined to suggest starting with a much simpler task. Pick a
> > discipline (preferably one with a professional society who can tell your
> > their estimate of current male/female ratio over (say) the past 5 years),
> > limit the Wikipedia articles to topics in that discipline, and limit the
> > citations to those published within the last 5 years. Indeed, perhaps
> > limiting it to publications that are principally from the same country(s)
> > as the professional society from which you get the data (as clearly
> > men/women's participation in any discipline can vary with different
> > countries for cultural reasons). Then you have some way to gauge whether
> > Wikipedia is showing more or less gender bias in its citations than the
> > discipline itself exhibits through publication. Quite a challenge!
> >
> > And of course, it is not Wikipedia that adds citations. It is individual
> > contributor who add citations. Does the sex/gender of the contributor
> have
> > any correlation to any observed bias? Again, the task is made more
> > difficult because a lot of Wikipedians don't identify their sex/gender.
> >
> > The other thing to be alert to is the difference in how (I believe)
> > Wikipedians cite compared to researchers. As a researcher, I will of
> course
> > be reading papers in my field all the time and what I read will influence
> > my subsequent work. Therefore when I write about my research, my
> citations
> > are referring to papers that I have already read and whose authors may be
> > familiar to me from their other work, having met them at a conferences,
> > private correspondence, etc. However as a Wikipedian, I am only partially
> > operating that way (mostly when I write new articles or significantly
> > expand them, that is, when I am doing the research). A lot of the time I
> am
> > adding citations relating to content other people (often new users) have
> > added/changed without citation. These come up on my watchlist all the
> time.
> > What do I do? Of course I could revert saying "no citation provided", but
> > that's not the way to encourage new contributors nor to grow the
> > encyclopedia, so if the information seems plausible (not obviously
> > vandalism), I will attempt to find a citation for it (using tools like
> > Google and other topic-specialise search tools). This is what I call
> "lucky
> > dip" mode of citing as obviously I have no idea what the source was for
> the
> > original contributor. The sources I find from my search may not already
> be
> > known to me (frequently they are not). Or to summarise, IMHO, researchers
> > (or Wikipedians in "new content mode") cite a source already known to
> them
> > and whose authors may be known to them and could consciously or
> > unconsciously engage in some discrimination in citation based on
> sex/gender
> > or other criteria, whereas Wikipedians in "updating mode" are likely to
> be
> > citing a source not previously known to them and may be happy just to
> have
> > found a source and are unlikely to be spending a lot of their time
> > researching the authors of that source to be extent they could then
> > consciously or unconsciously exercise discrimination on sex/gender. If I
> > invest any extra effort in such a situations, it's probably because the
> > wording of the source is a close match to the Wikipedia article which
> begs
> > the question of copyright violation (which needs to be dealt with by
> > deletion or rewriting) or being a Wikipedia mirror (which is obviously
> not
> > an acceptable citation).
> >
> > So I suspect whether a citation was added by the same contributor as the
> > content it supports or a subsequent contributor probably makes a
> difference
> > to the likelihood of conscious/unconscious discrimination.
> >
> > Also, finally, often Wikipedia cites web pages and other sources that do
> > not have any individual authorship, e.g. government websites. Remember
> that
> > Wikipedia prefers open citations over paywalled citations and a lot of
> the
> > publications behind paywalls are individually authored.
> >
> > Your proposed research has a lot of interesting challenges and a number
> of
> > limitations. I'm not saying don't do it, but I am saying start very small
> > and see if you can find any evidence to support your hypothesis before
> > embarking on a larger study. Because contributor behaviour is what you
> are
> > trying to study, you probably need to do both quantitative and
> qualitative
> > experiments. E.g. I have described the two modes of citation I do, but I
> > cannot say how typical my behaviour is.
> >
> > Kerry
> >
> > -----Original Message-----
> > From: Wiki-research-l [mailto:
> [hidden email]]
> > On Behalf Of Leila Zia
> > Sent: Friday, 23 August 2019 3:44 AM
> > To: Research into Wikimedia content and communities <
> > [hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> >
> > Hi Greg,
> >
> > A few comments if you're going to go with "proportion of male vs female
> > authors of the source material used as citations in arbitrary
> > articles":
> >
> > * Please differentiate between sex (female, male, ...) and gender (woman,
> > man, ...). My understanding from your initial email is that you want to
> > stay focused on gender, not sex.
> >
> > * Unless you have reliable sources about the gender of an author, I would
> > not recommend trying to predict what the gender is. (As you may know,
> this
> > is not uncommon in social media studies, for example, to predict the
> gender
> > of the author based on their image or their name.
> > These approaches introduce biases and social challenges.)
> >
> > * Re your question about whether WMF has resources to look into this
> > question in-house: I can't speak for the whole of WMF, however, I can
> share
> > more about the Research team's direction. As part of our future work, we
> > would like to "help contributors monitor violations of core content
> > policies and assess information reliability and bias both granularly and
> at
> > scale". [1] The question you proposed can fall under assessing bias in
> > content (considering citations as part of the content). I expect us to
> > focus first on the piece about violations of core content policies and
> > information reliability and come back to the bias question later. As a
> > result, we won't have bandwidth to do your proposal in-house at the
> moment.
> > Sorry about that.
> >
> > I hope this helps.
> >
> > Best,
> > Leila
> >
> > [1] Section 2 of our Knowledge Integrity whitepaper:
> >
> >
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
> >
> >
> > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > > mostly along the lines of your first bullet point, but I'd be
> > > interested in research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM
> > > <[hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more specific
> > > > than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > > --------------------------------------------------------------------
> > > > --
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> > > > is also an initiative to place citations in an annotated structured
> > > > repository (wikicite). I do not know the current status of wikicite
> > > > or if/when this could be used for this inquiry--either to examine
> > > > all, or a sensible subset of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary and urgent. The balance could be better, the same, or
> > > > worse than the citation balances we already know, and the scale of
> the
> > effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > > > Does the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender
> > > > balance of citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used
> > > > as citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men vs
> > women?
> > > > * the quality/quantity of citations in articles that are gendered by
> > > > some other criteria (e.g. reader interest, romantic comedy vs action
> > film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l
> > > > [mailto:[hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> > > > is also an initiative to place citations in an annotated structured
> > > > repository (wikicite). I do not know the current status of wikicite
> > > > or if/when this could be used for this inquiry--either to examine
> > > > all, or a sensible subset of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary and urgent. The balance could be better, the same, or
> > > > worse than the citation balances we already know, and the scale of
> the
> > effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > > > Does the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 13
> > ************************************************
> >
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 14
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Kerry Raymond
Hmm. I get the error that the mailing list doesn't exist. But if you write here

https://meta.wikimedia.org/wiki/Talk:WikiCite

someone should be able to point you in the right direction. There has been activity there within the last month.

Kerry

-----Original Message-----
From: Wiki-research-l [mailto:[hidden email]] On Behalf Of Greg
Sent: Saturday, 24 August 2019 7:13 AM
To: [hidden email]
Subject: Re: [Wiki-research-l] gender balance of wikipedia citations

Hi all-

One more thing: on twitter, I was advised that this list and the wikicite
google group were the best places to discuss research around citations.
Although I would like to post about this line of inquiry to the wikicite
group, it appears to be a private group. As an outsider, I have not been
able to access/view the group content/or even see who owns the group and is
the correct person to contact. I have mentioned that I can not access the
group to the wikicite twitter handle, and received a 'like' (?) but no
response, and nothing has changed.

If you are a member of both lists, would you be willing to point to this
thread from the wikicite group? If the group is not open to the public, at
least the ideas will be there.

Many thanks,

Greg

On Fri, Aug 23, 2019 at 12:01 AM <
[hidden email]> wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. sockpuppets and how to find them sooner (Kerry Raymond)
>    2. Re: sockpuppets and how to find them sooner (Kerry Raymond)
>    3. Re: sockpuppets and how to find them sooner (RhinosF1)
>    4. Re: gender balance of wikipedia citations (Greg)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 23 Aug 2019 15:57:15 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID: <001b01d55977$9f51dbe0$ddf593a0$@gmail.com>
> Content-Type: text/plain;       charset="us-ascii"
>
> Currently, to open a sockpuppet investigation, you must name the two (or
> more) accounts that you believe to be sockpuppets with "clear, behavioural
> evidence of sock puppetry" which is typically in the form of pairs of edits
> that demonstrate similar edit behaviours that are unlikely to naturally
> occur. Now if you spend enough time on-wiki, you develop an intuition about
> behaviours you see on your watchlist and in article edit histories. Often I
> am highly suspicious that an account is a sockpuppet, but I cannot report
> them because I don't know which other account is involved.
>
>
>
> As a example, I recently encounted User:Shelati an account about 1 day old
> at that time with nearly 100 edits in that day all about 1-2 minutes apart,
> mostly making a similar change to a large number of Australian place
> infoboxes.
>
>
>
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati
> <
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&of
> fset=20190728053057&limit=100&target=Shelati
> <https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&offset=20190728053057&limit=100&target=Shelati>
> >
> &offset=20190728053057&limit=100&target=Shelati
>
>
>
> Genuine new users do not edit that quickly, do not use templates and do not
> mess structurally with infoboxes (at most they try to change the values).
> It
> "smelled" like a sockpuppet. However, as I did not recognise that pattern
> of
> edit behaviour as being that of any other user I was familiar with, it
> wasn't something I could report for sockpuppet investigation. Anyhow after
> about 2 weeks, the user was blocked as a sockpuppet. Someone must have
> noticed and figured out the other account:
>
>
>
>
> https://en.wikipedia.org/wiki/Wikipedia:Sockpuppet_investigations/Meganesia/
> Archive
>
>
>
> Two weeks and 1,279 edits later . that's over 1000 possibly problematic
> edits after I first suspected them. But that's nothing compared with
> another
> ongoing situation in which a very large number of different IPs are engaged
> in a pattern of problem edits on mostly Australian articles (a few
> different
> types of edits but an obvious "quack like a duck" situation). The IP number
> changes frequently (and one assumes deliberately). The edits potentially go
> back to 2013 but appear to have intensified in 2018/2019. Here's one user's
> summary of all the IP addresses involved, and the extent to which they have
> been cleaned up, given many thousands of edits are involved, see:
>
>
>
> https://en.wikipedia.org/wiki/User:IamNotU/History_cleanup
>
>
>
> As well as the damage done to the content (which harms the readers), these
> IP sockpuppets are consuming enormous amounts of effort to track them down
> and revert them, which could be more productively used to improve the
> content. We need better tools to foil these pests. So I want to put that
> challenge out to this list.
>
>
>
> Kerry
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 23 Aug 2019 16:26:43 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID: <002001d5597b$bd1f6b20$375e4160$@gmail.com>
> Content-Type: text/plain;       charset="us-ascii"
>
> To reply to my own question .
>
>
>
> Can we find a way to create a "signature" of an account's pattern of
> editing? Perhaps it might be a set of signatures, maybe one for the
> categories that the account appears to be active in, another for the type
> of
> edit, etc. Then if these signatures were calculated for all banned accounts
> or currently blocked accounts (or at least ones with a long enough
> contribution history to make it worthwhile - we're not interested in
> one-edit vandals), then we could have a tool that could be run to quickly
> compare one account against the signatures of banned/blocked accounts as
> well as the cumulative edits of a set of known sockpuppets (i.e. treat them
> as a single account) to determine if this may be a sockpuppet case meriting
> further investigation. I imagine that it would be too expensive
> computationally to actually run comparisons of the contribution histories
> of
> all "bad guy" accounts against the suspicious account which is why I
> propose
> a "signature" approach (but I'm happy to be told otherwise).
>
>
>
> If we had such a tool and it proves reasonably reliable in identifying
> likely sockpuppets (not asking for guarantees but close enough not to be a
> waste of time to investigate), then we could routinely use it on new
> accounts or reactivating accounts (i.e. possible sleeper accounts) once
> they
> have a long enough editing history to enable the tool to operate
> effectively
> to provide automated early warning of new/reactivating accounts appearing
> suspiciously similar to "bad guy" accounts.
>
>
>
> But this is a hard problem, both technically and socially (Assume Good
> Faith, Privacy, etc), so I welcome the thoughts of others.
>
>
>
> Kerry
>
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 23 Aug 2019 07:43:29 +0100
> From: RhinosF1 <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>, [hidden email]
> Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID:
>         <CAK3HvEMj8VCPu+uVUmPW+RepJ6E-OULaLPA15ALv9C=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Just a note that you can still go through warnings for vandalism etc. and
> report to AIV.
>
> Or at that edit speed, you may have a chance at AN at reporting for
> bot-like edits which will draw attention to the account.
>
> If you ever need help, things like #wikipedia-en-help on Freenode IRC exist
> so you can ask other users.
>
> RhinosF1
> Miraheze Volunteer
>
> On Fri, 23 Aug 2019 at 06:57, Kerry Raymond <[hidden email]>
> wrote:
>
> > Currently, to open a sockpuppet investigation, you must name the two (or
> > more) accounts that you believe to be sockpuppets with "clear,
> behavioural
> > evidence of sock puppetry" which is typically in the form of pairs of
> edits
> > that demonstrate similar edit behaviours that are unlikely to naturally
> > occur. Now if you spend enough time on-wiki, you develop an intuition
> about
> > behaviours you see on your watchlist and in article edit histories.
> Often I
> > am highly suspicious that an account is a sockpuppet, but I cannot report
> > them because I don't know which other account is involved.
> >
> >
> >
> > As a example, I recently encounted User:Shelati an account about 1 day
> old
> > at that time with nearly 100 edits in that day all about 1-2 minutes
> apart,
> > mostly making a similar change to a large number of Australian place
> > infoboxes.
> >
> >
> >
> > https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati
> > <
> >
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&of
> > fset=20190728053057&limit=100&target=Shelati
> > <
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&offset=20190728053057&limit=100&target=Shelati
> >
> > >
> > &offset=20190728053057&limit=100&target=Shelati
> >
> >
> >
> > Genuine new users do not edit that quickly, do not use templates and do
> not
> > mess structurally with infoboxes (at most they try to change the values).
> > It
> > "smelled" like a sockpuppet. However, as I did not recognise that pattern
> > of
> > edit behaviour as being that of any other user I was familiar with, it
> > wasn't something I could report for sockpuppet investigation. Anyhow
> after
> > about 2 weeks, the user was blocked as a sockpuppet. Someone must have
> > noticed and figured out the other account:
> >
> >
> >
> >
> >
> https://en.wikipedia.org/wiki/Wikipedia:Sockpuppet_investigations/Meganesia/
> > Archive
> >
> >
> >
> > Two weeks and 1,279 edits later . that's over 1000 possibly problematic
> > edits after I first suspected them. But that's nothing compared with
> > another
> > ongoing situation in which a very large number of different IPs are
> engaged
> > in a pattern of problem edits on mostly Australian articles (a few
> > different
> > types of edits but an obvious "quack like a duck" situation). The IP
> number
> > changes frequently (and one assumes deliberately). The edits potentially
> go
> > back to 2013 but appear to have intensified in 2018/2019. Here's one
> user's
> > summary of all the IP addresses involved, and the extent to which they
> have
> > been cleaned up, given many thousands of edits are involved, see:
> >
> >
> >
> > https://en.wikipedia.org/wiki/User:IamNotU/History_cleanup
> >
> >
> >
> > As well as the damage done to the content (which harms the readers),
> these
> > IP sockpuppets are consuming enormous amounts of effort to track them
> down
> > and revert them, which could be more productively used to improve the
> > content. We need better tools to foil these pests. So I want to put that
> > challenge out to this list.
> >
> >
> >
> > Kerry
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> --
> RhinosF1
> Miraheze Volunteer
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 23 Aug 2019 00:01:15 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Wow, Kerry! Thank you for taking the time to write all these thoughts out.
>
> I'm asking the question because I'm concerned that the gender balance of
> the authors being cited on wikipedia is different from the already quite
> bad patterns in academia. My fear is that the citation gender imbalance on
> Wikipedia is more pronounced. If so, it is not just perpetuating the
> problem, but making it worse by surfacing certain authors and ideas even
> more frequently, or hardly at all. I would like to know if this is the
> case, and if so, how big the effect is.
>
> In my last message, I mention a study about a set of award-winning
> political science books (the researchers study the citation gender
> imbalance for that set). I just saw this study today, but I began to think
> that it/the set of works--or some similar set of titles--could possibly be
> a good place to begin, especially if the original researchers were willing
> to share the list of titles/authors/gender/etc that they put
> together/worked with. Then it seems it would mostly be a matter of figuring
> out how to understand how those titles are cited on Wikipedia--through
> either the citation dataset or wikicite--to see if/how the citation
> patterns differ (i.e., if the works by women/men are cited more
> frequently/at the same rate/less frequently on Wikipedia than what the
> researchers found in the original study).
>
> This seems like it would be easier to do than what you propose, but perhaps
> the idea is not sound. Until very recently, I thought I could find the
> answer in an existing paper! I honestly don't know the best way to get the
> answer, but I would like to know the answer and think it's important to
> look at.
>
> All of the things you bring up--from the gender of the editor, to the type
> of editing being done, to the issues around multiple authors/paywalls/year
> of publication/field--complicate the inquiry, and in particular a larger
> one. I agree with what you say about doing something small first to see
> what's there.
>
> Thanks again for all your thoughts.
> Greg
>
>
>
> On Thu, Aug 22, 2019 at 9:41 PM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 22 Aug 2019 18:47:48 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Leila,
> >
> > Thanks for your thoughts.
> >
> > Having just read Troy Vettese's very powerful essay, Sexism in the
> Academy
> > (
> > https://nplusonemag.com/issue-34/essays/sexism-in-the-academy/), I wish
> > this were a top priority.
> >
> > I stumbled upon a study today--it came up in the Washington Post's
> > excellent series on gender bias in political science. The authors look
> at a
> > set of award winning political science books and the gender imbalance in
> > the citations drawn from google scholar.  I'm linking the piece here in
> > case anyone on this list is interested now, or in the future, in how the
> > patterns on Wikipedia compare.
> >
> > Washington Post piece: "There’s a gender gap in who wins political
> science
> > book awards – and in how widely they’re cited"
> >
> >
> https://www.washingtonpost.com/politics/2019/08/22/theres-gender-gap-who-wins-political-science-book-awards-how-widely-theyre-cited/
> > "Just as significantly, women’s award-winning books receive fewer
> scholarly
> > citations than men’s award-winning volumes — and this disparity has
> grown,
> > rather than shrunk, in recent years. Over the entire period, APSA
> > award-winning volumes by women averaged 43 percent fewer citations per
> year
> > than those by male authors."
> >
> > Paper: "Winning awards and gaining recognition: An impact analysis of
> APSA
> > section book prizes"
> > https://www.sciencedirect.com/science/article/abs/pii/S0362331918300867
> >
> >
> > Best,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 3:44 PM <
> > [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. Re: gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Leila Zia)
> > >    3. Wikimania 2019 disinformation meetup follow-up (Leila Zia)
> > >    4. Upcoming Research Newsletter (special issue on gender gap
> > >       research): New papers open for review (Mohammed Sadat Abdulai)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Thu, 22 Aug 2019 09:57:15 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <CAOO9DNuSYzzaVwcdqiWA7pj671z3N43XOSwv6DtW0SxWg=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > mostly
> > > along the lines of your first bullet point, but I'd be interested in
> > > research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more specific
> > > > than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > >
> ----------------------------------------------------------------------
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> is
> > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > (wikicite). I do not know the current status of wikicite or if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender balance
> > of
> > > > citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used as
> > > > citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men vs
> > > women?
> > > > * the quality/quantity of citations in articles that are gendered by
> > some
> > > > other criteria (e.g. reader interest, romantic comedy vs action
> film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l [mailto:
> > > [hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> is
> > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > (wikicite). I do not know the current status of wikicite or if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 10:43:51 -0700
> > > From: Leila Zia <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <CAK0Oe2uCo70_=ma2b=2d+fvr4GseEVxOP0sh=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Greg,
> > >
> > > A few comments if you're going to go with "proportion of male vs
> > > female authors of the source material used as citations in arbitrary
> > > articles":
> > >
> > > * Please differentiate between sex (female, male, ...) and gender
> > > (woman, man, ...). My understanding from your initial email is that
> > > you want to stay focused on gender, not sex.
> > >
> > > * Unless you have reliable sources about the gender of an author, I
> > > would not recommend trying to predict what the gender is. (As you may
> > > know, this is not uncommon in social media studies, for example, to
> > > predict the gender of the author based on their image or their name.
> > > These approaches introduce biases and social challenges.)
> > >
> > > * Re your question about whether WMF has resources to look into this
> > > question in-house: I can't speak for the whole of WMF, however, I can
> > > share more about the Research team's direction. As part of our future
> > > work, we would like to "help contributors monitor violations of core
> > > content policies and assess information reliability and bias both
> > > granularly and at scale". [1] The question you proposed can fall under
> > > assessing bias in content (considering citations as part of the
> > > content). I expect us to focus first on the piece about violations of
> > > core content policies and information reliability and come back to the
> > > bias question later. As a result, we won't have bandwidth to do your
> > > proposal in-house at the moment. Sorry about that.
> > >
> > > I hope this helps.
> > >
> > > Best,
> > > Leila
> > >
> > > [1] Section 2 of our Knowledge Integrity whitepaper:
> > >
> > >
> >
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
> > >
> > >
> > > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]>
> wrote:
> > > >
> > > > Hi Kerry,
> > > > Those are all very interesting ways to look at this. I was thinking
> > > mostly
> > > > along the lines of your first bullet point, but I'd be interested in
> > > > research in any of those areas.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Thu, Aug 22, 2019 at 5:00 AM <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Send Wiki-research-l mailing list submissions to
> > > > >         [hidden email]
> > > > >
> > > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > > >
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > > or, via email, send a message with subject or body 'help' to
> > > > >         [hidden email]
> > > > >
> > > > > You can reach the person managing the list at
> > > > >         [hidden email]
> > > > >
> > > > > When replying, please edit your Subject line so it is more specific
> > > > > than "Re: Contents of Wiki-research-l digest..."
> > > > >
> > > > >
> > > > > Today's Topics:
> > > > >
> > > > >    1. gender balance of wikipedia citations (Greg)
> > > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > > >
> > > > >
> > > > >
> > ----------------------------------------------------------------------
> > > > >
> > > > > Message: 1
> > > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > > From: Greg <[hidden email]>
> > > > > To: [hidden email]
> > > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > > Message-ID:
> > > > >         <
> > > > > [hidden email]
> >
> > > > > Content-Type: text/plain; charset="UTF-8"
> > > > >
> > > > > Greetings!
> > > > >
> > > > > I was looking for information about the gender balance of Wikipedia
> > > > > citations and no one I've asked knows of any work on this topic. Do
> > > you?
> > > > >
> > > > > I think this is an important question.
> > > > >
> > > > > Here's what I've learned so far:
> > > > >
> > > > > Wikipedia citations are currently in the form of text strings.
> There
> > is
> > > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > > (wikicite). I do not know the current status of wikicite or if/when
> > > this
> > > > > could be used for this inquiry--either to examine all, or a
> sensible
> > > subset
> > > > > of the citations.
> > > > >
> > > > > My perspective is that understanding the gender balance is
> necessary
> > > and
> > > > > urgent. The balance could be better, the same, or worse than the
> > > citation
> > > > > balances we already know, and the scale of the effect is quite
> large.
> > > > >
> > > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > > interested in pursuing? If so, what is the best way to get started?
> > > Does
> > > > > the WMF have the resources and interest to look into this matter
> > > inhouse?
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > Greg
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > Message: 2
> > > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > > From: "Kerry Raymond" <[hidden email]>
> > > > > To: "'Research into Wikimedia content and communities'"
> > > > >         <[hidden email]>
> > > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> citations
> > > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > > Content-Type: text/plain;       charset="UTF-8"
> > > > >
> > > > > Could you elaborate a bit more on what you mean by the gender
> balance
> > > of
> > > > > citations?
> > > > >
> > > > > Are you talking about:
> > > > >
> > > > > * proportion of male vs female authors of the source material used
> as
> > > > > citations in arbitrary articles>
> > > > > *  the quality/quantity of citations in biography articles of men
> vs
> > > women?
> > > > > * the quality/quantity of citations in articles that are gendered
> by
> > > some
> > > > > other criteria (e.g. reader interest, romantic comedy vs action
> > film)?
> > > > >
> > > > > Kerry
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wiki-research-l [mailto:
> > > [hidden email]]
> > > > > On Behalf Of Greg
> > > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > > To: [hidden email]
> > > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > >
> > > > > Greetings!
> > > > >
> > > > > I was looking for information about the gender balance of Wikipedia
> > > > > citations and no one I've asked knows of any work on this topic. Do
> > > you?
> > > > >
> > > > > I think this is an important question.
> > > > >
> > > > > Here's what I've learned so far:
> > > > >
> > > > > Wikipedia citations are currently in the form of text strings.
> There
> > is
> > > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > > (wikicite). I do not know the current status of wikicite or if/when
> > > this
> > > > > could be used for this inquiry--either to examine all, or a
> sensible
> > > subset
> > > > > of the citations.
> > > > >
> > > > > My perspective is that understanding the gender balance is
> necessary
> > > and
> > > > > urgent. The balance could be better, the same, or worse than the
> > > citation
> > > > > balances we already know, and the scale of the effect is quite
> large.
> > > > >
> > > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > > interested in pursuing? If so, what is the best way to get started?
> > > Does
> > > > > the WMF have the resources and interest to look into this matter
> > > inhouse?
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > Greg
> > > > > _______________________________________________
> > > > > Wiki-research-l mailing list
> > > > > [hidden email]
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > Subject: Digest Footer
> > > > >
> > > > > _______________________________________________
> > > > > Wiki-research-l mailing list
> > > > > [hidden email]
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > > ************************************************
> > > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 3
> > > Date: Thu, 22 Aug 2019 13:36:17 -0700
> > > From: Leila Zia <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: [Wiki-research-l] Wikimania 2019 disinformation meetup
> > >         follow-up
> > > Message-ID:
> > >         <CAK0Oe2sodYJpkuhSqgo3dtfDr=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi,
> > >
> > > This message is for those of you who attended the disinformation
> > > meet-up [0] in Wikimania 2019 [1] or others who may be interested.
> > >
> > > * The notes from our meet-up are now posted in the bottom of the page
> > [0].
> > >
> > > * I was tasked to see if space.wmflabs.org is the place for us to
> > > continue conversations about this topic. The answer is yes. Thanks to
> > > the help of Elena Lappen, we now have a dedicated subcategory for
> > > disinformation:
> > > https://discuss-space.wmflabs.org/c/research/disinformation . Feel
> > > free to subscribe, watch, and/or post new topics if you're involved in
> > > this space.
> > >
> > > * If you are new to this conversation, please read the purpose of the
> > > subcategory at
> > >
> >
> https://discuss-space.wmflabs.org/t/about-the-disinformation-category/949
> > > and welcome! :)
> > >
> > > Best,
> > > Leila
> > >
> > > [0] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation
> > > [1] https://wikimania.wikimedia.org/wiki/2019:Program
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 4
> > > Date: Thu, 22 Aug 2019 22:43:53 +0000 (UTC)
> > > From: Mohammed Sadat Abdulai <[hidden email]>
> > > To: Research Into Wikimedia Content and Communities
> > >         <[hidden email]>
> > > Subject: [Wiki-research-l] Upcoming Research Newsletter (special issue
> > >         on gender gap research): New papers open for review
> > > Message-ID: <[hidden email]>
> > > Content-Type: text/plain; charset=UTF-8
> > >
> > >  Hi everyone,
> > > We’re preparing for the August 2019 research newsletter and looking for
> > > contributors. Please take a look at
> > > https://etherpad.wikimedia.org/p/WRN201908 and add your name next to
> any
> > > paper you are interested in covering. Our target publication date is on
> > 31
> > > August 11:59 UTC. As usual, short notes and one-paragraph reviews are
> > most
> > > welcome.
> > >  For the August edition, we are planning a special issue focusing
> mainly
> > > on recent gender gap/gender bias research. (Upcoming special issues
> > topics
> > > may include health and education.) There are about 20 papers from this
> > area
> > > on our todo list which will all be covered in the August issue, either
> > as a
> > > mere list item or - with your help - in form of a more informative
> > writeup
> > > or review. They include:
> > >    - Analyzing Gender Stereotyping in Bollywood Movies
> > >
> > >    - Breaking the glass ceiling on Wikipedia| journal
> > >
> > >    - Breastfeeding, Authority, and Genre: Women's Ethos in Wikipedia
> and
> > > Blogs
> > >
> > >    - Cyberfeminism on Wikipedia: Visibility and deliberation in
> feminist
> > > Wikiprojects
> > >
> > >    - Gender and deletion on Wikipedia
> > >
> > >    - Gender imbalance and Wikipedia
> > >
> > >    - Gender Markers in Wikipedia Usernames
> > >
> > >    - How do students trust Wikipedia? An examination across genders
> > >
> > >    - Investigating the Gender Pronoun Gap in Wikipedia
> > >
> > >    - It’s Not What You Think: Gender Bias in Information about Fortune
> > > 1000 CEOs on Wikipedia
> > >
> > >    - Mapping and Bridging the Gender Gap: An Ethnographic Study of
> Indian
> > > Wikipedians and Their Motivations to Contribute
> > >
> > >    - People Who Can Take It: How Women Wikipedians Negotiate and
> Navigate
> > > Safety
> > >
> > >    - Redressing Gender Inequities on Wikipedia Through an Editathon
> > >
> > >    - Similar Gaps, Different Origins? Women Readers and Editors at
> Greek
> > > Wikipedia
> > >
> > >    - Simulation Experiments on (the Absence of) Ratings Bias in
> > Reputation
> > > Systems
> > >
> > >    - The Gendered Presentation of Professions on Wikipedia
> > >
> > >    - Who Counts as a Notable Sociologist on Wikipedia? Gender, Race,
> and
> > > the “Professor Test”
> > >
> > >    - Who Wants to Read This?: A Method for Measuring Topical
> > > Representativeness in User Generated Content Systems
> > >
> > >    - Women and Wikipedia. Diversifying Editors and Enhancing Content
> > > through Library Edit-a-Thons
> > >
> > > Masssly and Tilman Bayer
> > >
> > > [1] Research:Newsletter - Meta[2] WikiResearch (@WikiResearch) on
> Twitter
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 12
> > > ************************************************
> > >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Fri, 23 Aug 2019 14:41:09 +1000
> > From: "Kerry Raymond" <[hidden email]>
> > To: "'Research into Wikimedia content and communities'"
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <001001d5596c$fe22a100$fa67e300$@gmail.com>
> > Content-Type: text/plain;       charset="utf-8"
> >
> > Yes, that was my thought. It would be difficult to know the sex (or the
> > gender) of an author name on a paper. There would inevitably be a lot
> that
> > you could not determine. And certainly in the sciences multi-author pages
> > are the norm and even where you did know the sex/gender of all, do you
> > assign some part-score? E.g. 0 for all male, 1 for all female, 0.6 for 3
> > women and 2 men.
> >
> > But I am curious why you are asking the question? That the
> > writing/research of women is under-represented in Wikipedia citations? If
> > so, without conducting any research, I'd say "yes it is
> under-represented".
> > But my reason would be because women are under-represented as
> > writers/researchers in the first place.  And certainly the older the
> > source, the more likely it is to be written by a man. So to investigate
> > gender bias in citations in Wikipedia, you would have to estimate the
> > proportion of men/women (or at least their outputs) over time in a given
> > discipline and then ask the question, "taking into account of the time of
> > publication of a citation and the proportion of men/women active in this
> > discipline at that time, do Wikipedia citations show a sex/gender
> basis?".
> > Hmm ... very tricky.
> >
> > I'd be inclined to suggest starting with a much simpler task. Pick a
> > discipline (preferably one with a professional society who can tell your
> > their estimate of current male/female ratio over (say) the past 5 years),
> > limit the Wikipedia articles to topics in that discipline, and limit the
> > citations to those published within the last 5 years. Indeed, perhaps
> > limiting it to publications that are principally from the same country(s)
> > as the professional society from which you get the data (as clearly
> > men/women's participation in any discipline can vary with different
> > countries for cultural reasons). Then you have some way to gauge whether
> > Wikipedia is showing more or less gender bias in its citations than the
> > discipline itself exhibits through publication. Quite a challenge!
> >
> > And of course, it is not Wikipedia that adds citations. It is individual
> > contributor who add citations. Does the sex/gender of the contributor
> have
> > any correlation to any observed bias? Again, the task is made more
> > difficult because a lot of Wikipedians don't identify their sex/gender.
> >
> > The other thing to be alert to is the difference in how (I believe)
> > Wikipedians cite compared to researchers. As a researcher, I will of
> course
> > be reading papers in my field all the time and what I read will influence
> > my subsequent work. Therefore when I write about my research, my
> citations
> > are referring to papers that I have already read and whose authors may be
> > familiar to me from their other work, having met them at a conferences,
> > private correspondence, etc. However as a Wikipedian, I am only partially
> > operating that way (mostly when I write new articles or significantly
> > expand them, that is, when I am doing the research). A lot of the time I
> am
> > adding citations relating to content other people (often new users) have
> > added/changed without citation. These come up on my watchlist all the
> time.
> > What do I do? Of course I could revert saying "no citation provided", but
> > that's not the way to encourage new contributors nor to grow the
> > encyclopedia, so if the information seems plausible (not obviously
> > vandalism), I will attempt to find a citation for it (using tools like
> > Google and other topic-specialise search tools). This is what I call
> "lucky
> > dip" mode of citing as obviously I have no idea what the source was for
> the
> > original contributor. The sources I find from my search may not already
> be
> > known to me (frequently they are not). Or to summarise, IMHO, researchers
> > (or Wikipedians in "new content mode") cite a source already known to
> them
> > and whose authors may be known to them and could consciously or
> > unconsciously engage in some discrimination in citation based on
> sex/gender
> > or other criteria, whereas Wikipedians in "updating mode" are likely to
> be
> > citing a source not previously known to them and may be happy just to
> have
> > found a source and are unlikely to be spending a lot of their time
> > researching the authors of that source to be extent they could then
> > consciously or unconsciously exercise discrimination on sex/gender. If I
> > invest any extra effort in such a situations, it's probably because the
> > wording of the source is a close match to the Wikipedia article which
> begs
> > the question of copyright violation (which needs to be dealt with by
> > deletion or rewriting) or being a Wikipedia mirror (which is obviously
> not
> > an acceptable citation).
> >
> > So I suspect whether a citation was added by the same contributor as the
> > content it supports or a subsequent contributor probably makes a
> difference
> > to the likelihood of conscious/unconscious discrimination.
> >
> > Also, finally, often Wikipedia cites web pages and other sources that do
> > not have any individual authorship, e.g. government websites. Remember
> that
> > Wikipedia prefers open citations over paywalled citations and a lot of
> the
> > publications behind paywalls are individually authored.
> >
> > Your proposed research has a lot of interesting challenges and a number
> of
> > limitations. I'm not saying don't do it, but I am saying start very small
> > and see if you can find any evidence to support your hypothesis before
> > embarking on a larger study. Because contributor behaviour is what you
> are
> > trying to study, you probably need to do both quantitative and
> qualitative
> > experiments. E.g. I have described the two modes of citation I do, but I
> > cannot say how typical my behaviour is.
> >
> > Kerry
> >
> > -----Original Message-----
> > From: Wiki-research-l [mailto:
> [hidden email]]
> > On Behalf Of Leila Zia
> > Sent: Friday, 23 August 2019 3:44 AM
> > To: Research into Wikimedia content and communities <
> > [hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> >
> > Hi Greg,
> >
> > A few comments if you're going to go with "proportion of male vs female
> > authors of the source material used as citations in arbitrary
> > articles":
> >
> > * Please differentiate between sex (female, male, ...) and gender (woman,
> > man, ...). My understanding from your initial email is that you want to
> > stay focused on gender, not sex.
> >
> > * Unless you have reliable sources about the gender of an author, I would
> > not recommend trying to predict what the gender is. (As you may know,
> this
> > is not uncommon in social media studies, for example, to predict the
> gender
> > of the author based on their image or their name.
> > These approaches introduce biases and social challenges.)
> >
> > * Re your question about whether WMF has resources to look into this
> > question in-house: I can't speak for the whole of WMF, however, I can
> share
> > more about the Research team's direction. As part of our future work, we
> > would like to "help contributors monitor violations of core content
> > policies and assess information reliability and bias both granularly and
> at
> > scale". [1] The question you proposed can fall under assessing bias in
> > content (considering citations as part of the content). I expect us to
> > focus first on the piece about violations of core content policies and
> > information reliability and come back to the bias question later. As a
> > result, we won't have bandwidth to do your proposal in-house at the
> moment.
> > Sorry about that.
> >
> > I hope this helps.
> >
> > Best,
> > Leila
> >
> > [1] Section 2 of our Knowledge Integrity whitepaper:
> >
> >
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrity_-_Wikimedia_Research_2030.pdf
> >
> >
> > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > > mostly along the lines of your first bullet point, but I'd be
> > > interested in research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM
> > > <[hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more specific
> > > > than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > > --------------------------------------------------------------------
> > > > --
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> > > > is also an initiative to place citations in an annotated structured
> > > > repository (wikicite). I do not know the current status of wikicite
> > > > or if/when this could be used for this inquiry--either to examine
> > > > all, or a sensible subset of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary and urgent. The balance could be better, the same, or
> > > > worse than the citation balances we already know, and the scale of
> the
> > effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > > > Does the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender
> > > > balance of citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used
> > > > as citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men vs
> > women?
> > > > * the quality/quantity of citations in articles that are gendered by
> > > > some other criteria (e.g. reader interest, romantic comedy vs action
> > film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l
> > > > [mailto:[hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of Wikipedia
> > > > citations and no one I've asked knows of any work on this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings. There
> > > > is also an initiative to place citations in an annotated structured
> > > > repository (wikicite). I do not know the current status of wikicite
> > > > or if/when this could be used for this inquiry--either to examine
> > > > all, or a sensible subset of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary and urgent. The balance could be better, the same, or
> > > > worse than the citation balances we already know, and the scale of
> the
> > effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > > > Does the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 13
> > ************************************************
> >
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 14
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Kerry Raymond
In reply to this post by Greg-2
I am inclined to think that political science has more Point of View in it than say chemistry. I also suspect it has fewer authors per book/paper. So I can imagine that people citing political science literature may be more inclined to cherry pick the sources that support their own POV which may involve some gender bias in some way. I would think it less likely in chemistry to cherry pick sources (which is not to say there are no divided schools of thought in chemistry but it is a more experimental discipline with strong commitment to factual data and less to opinion).

But having said all that, whether and in what circumstances that the selection of sources in Wikipedia might be sex/gender biased, I honestly don't know. But if it manifests outside of Wikipedia as you suggest, then I would be very surprised if it wasn't replicated in Wikipedia to some extent. But I guess your question is whether is Wikipedia merely reflects the society it lives in (similar levels of gender bias) or whether there is something about Wikipedia which acerbates or ameliorates the situation? I am genuine curious what a small study would discover and agree that replicating (as much as possible) the existing study outside of Wikipedia) provides a good starting point. You might approach the authors of that study to see if they are willing to collaborate on such a project, either in design, data sharing or more fully. I look forward to seeing the results.

Kerry

-----Original Message-----
From: Wiki-research-l [mailto:[hidden email]] On Behalf Of Greg
Sent: Friday, 23 August 2019 5:01 PM
To: [hidden email]
Subject: Re: [Wiki-research-l] gender balance of wikipedia citations

Wow, Kerry! Thank you for taking the time to write all these thoughts out.

I'm asking the question because I'm concerned that the gender balance of the authors being cited on wikipedia is different from the already quite bad patterns in academia. My fear is that the citation gender imbalance on Wikipedia is more pronounced. If so, it is not just perpetuating the problem, but making it worse by surfacing certain authors and ideas even more frequently, or hardly at all. I would like to know if this is the case, and if so, how big the effect is.

In my last message, I mention a study about a set of award-winning political science books (the researchers study the citation gender imbalance for that set). I just saw this study today, but I began to think that it/the set of works--or some similar set of titles--could possibly be a good place to begin, especially if the original researchers were willing to share the list of titles/authors/gender/etc that they put together/worked with. Then it seems it would mostly be a matter of figuring out how to understand how those titles are cited on Wikipedia--through either the citation dataset or wikicite--to see if/how the citation patterns differ (i.e., if the works by women/men are cited more frequently/at the same rate/less frequently on Wikipedia than what the researchers found in the original study).

This seems like it would be easier to do than what you propose, but perhaps the idea is not sound. Until very recently, I thought I could find the answer in an existing paper! I honestly don't know the best way to get the answer, but I would like to know the answer and think it's important to look at.

All of the things you bring up--from the gender of the editor, to the type of editing being done, to the issues around multiple authors/paywalls/year of publication/field--complicate the inquiry, and in particular a larger one. I agree with what you say about doing something small first to see what's there.

Thanks again for all your thoughts.
Greg



On Thu, Aug 22, 2019 at 9:41 PM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. Re: gender balance of wikipedia citations (Greg)
>    2. Re: gender balance of wikipedia citations (Kerry Raymond)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 22 Aug 2019 18:47:48 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Hi Leila,
>
> Thanks for your thoughts.
>
> Having just read Troy Vettese's very powerful essay, Sexism in the
> Academy (
> https://nplusonemag.com/issue-34/essays/sexism-in-the-academy/), I
> wish this were a top priority.
>
> I stumbled upon a study today--it came up in the Washington Post's
> excellent series on gender bias in political science. The authors look
> at a set of award winning political science books and the gender
> imbalance in the citations drawn from google scholar.  I'm linking the
> piece here in case anyone on this list is interested now, or in the
> future, in how the patterns on Wikipedia compare.
>
> Washington Post piece: "There’s a gender gap in who wins political
> science book awards – and in how widely they’re cited"
>
> https://www.washingtonpost.com/politics/2019/08/22/theres-gender-gap-w
> ho-wins-political-science-book-awards-how-widely-theyre-cited/
> "Just as significantly, women’s award-winning books receive fewer
> scholarly citations than men’s award-winning volumes — and this
> disparity has grown, rather than shrunk, in recent years. Over the
> entire period, APSA award-winning volumes by women averaged 43 percent
> fewer citations per year than those by male authors."
>
> Paper: "Winning awards and gaining recognition: An impact analysis of
> APSA section book prizes"
> https://www.sciencedirect.com/science/article/abs/pii/S036233191830086
> 7
>
>
> Best,
> Greg
>
> On Thu, Aug 22, 2019 at 3:44 PM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Leila Zia)
> >    3. Wikimania 2019 disinformation meetup follow-up (Leila Zia)
> >    4. Upcoming Research Newsletter (special issue on gender gap
> >       research): New papers open for review (Mohammed Sadat Abdulai)
> >
> >
> > --------------------------------------------------------------------
> > --
> >
> > Message: 1
> > Date: Thu, 22 Aug 2019 09:57:15 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <CAOO9DNuSYzzaVwcdqiWA7pj671z3N43XOSwv6DtW0SxWg=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Kerry,
> > Those are all very interesting ways to look at this. I was thinking
> mostly
> > along the lines of your first bullet point, but I'd be interested in
> > research in any of those areas.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 5:00 AM <
> > [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >        
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more
> > > specific than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > >
> > >
> > > ------------------------------------------------------------------
> > > ----
> > >
> > > Message: 1
> > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <
> > > [hidden email]
> > > CAOO9DNtY+>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of
> > > Wikipedia citations and no one I've asked knows of any work on
> > > this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings.
> > > There is also an initiative to place citations in an annotated
> > > structured
> > repository
> > > (wikicite). I do not know the current status of wikicite or
> > > if/when
> this
> > > could be used for this inquiry--either to examine all, or a
> > > sensible
> > subset
> > > of the citations.
> > >
> > > My perspective is that understanding the gender balance is  
> > > necessary
> and
> > > urgent. The balance could be better, the same, or worse than the
> citation
> > > balances we already know, and the scale of the effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> Does
> > > the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > From: "Kerry Raymond" <[hidden email]>
> > > To: "'Research into Wikimedia content and communities'"
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> > > citations
> > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > Content-Type: text/plain;       charset="UTF-8"
> > >
> > > Could you elaborate a bit more on what you mean by the gender
> > > balance
> of
> > > citations?
> > >
> > > Are you talking about:
> > >
> > > * proportion of male vs female authors of the source material used
> > > as citations in arbitrary articles>
> > > *  the quality/quantity of citations in biography articles of men
> > > vs
> > women?
> > > * the quality/quantity of citations in articles that are gendered
> > > by
> some
> > > other criteria (e.g. reader interest, romantic comedy vs action film)?
> > >
> > > Kerry
> > >
> > > -----Original Message-----
> > > From: Wiki-research-l [mailto:
> > [hidden email]]
> > > On Behalf Of Greg
> > > Sent: Thursday, 22 August 2019 1:19 PM
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of
> > > Wikipedia citations and no one I've asked knows of any work on
> > > this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings.
> > > There is also an initiative to place citations in an annotated
> > > structured
> > repository
> > > (wikicite). I do not know the current status of wikicite or
> > > if/when
> this
> > > could be used for this inquiry--either to examine all, or a
> > > sensible
> > subset
> > > of the citations.
> > >
> > > My perspective is that understanding the gender balance is  
> > > necessary
> and
> > > urgent. The balance could be better, the same, or worse than the
> citation
> > > balances we already know, and the scale of the effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> Does
> > > the WMF have the resources and interest to look into this matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > ************************************************
> > >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 22 Aug 2019 10:43:51 -0700
> > From: Leila Zia <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <CAK0Oe2uCo70_=ma2b=2d+fvr4GseEVxOP0sh=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Greg,
> >
> > A few comments if you're going to go with "proportion of male vs
> > female authors of the source material used as citations in arbitrary
> > articles":
> >
> > * Please differentiate between sex (female, male, ...) and gender
> > (woman, man, ...). My understanding from your initial email is that
> > you want to stay focused on gender, not sex.
> >
> > * Unless you have reliable sources about the gender of an author, I
> > would not recommend trying to predict what the gender is. (As you
> > may know, this is not uncommon in social media studies, for example,
> > to predict the gender of the author based on their image or their name.
> > These approaches introduce biases and social challenges.)
> >
> > * Re your question about whether WMF has resources to look into this
> > question in-house: I can't speak for the whole of WMF, however, I
> > can share more about the Research team's direction. As part of our
> > future work, we would like to "help contributors monitor violations
> > of core content policies and assess information reliability and bias
> > both granularly and at scale". [1] The question you proposed can
> > fall under assessing bias in content (considering citations as part
> > of the content). I expect us to focus first on the piece about
> > violations of core content policies and information reliability and
> > come back to the bias question later. As a result, we won't have
> > bandwidth to do your proposal in-house at the moment. Sorry about that.
> >
> > I hope this helps.
> >
> > Best,
> > Leila
> >
> > [1] Section 2 of our Knowledge Integrity whitepaper:
> >
> >
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrit
> y_-_Wikimedia_Research_2030.pdf
> >
> >
> > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was
> > > thinking
> > mostly
> > > along the lines of your first bullet point, but I'd be interested
> > > in research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM <
> > [hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >        
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more
> > > > specific than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > >
> ----------------------------------------------------------------------
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]
> > > > CAOO9DNtY+om>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of
> > > > Wikipedia citations and no one I've asked knows of any work on
> > > > this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings.
> > > > There
> is
> > > > also an initiative to place citations in an annotated structured
> > repository
> > > > (wikicite). I do not know the current status of wikicite or
> > > > if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a
> > > > sensible
> > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  
> > > > necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community
> > > > is interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> > > > citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender
> > > > balance
> > of
> > > > citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material
> > > > used as citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of
> > > > men vs
> > women?
> > > > * the quality/quantity of citations in articles that are
> > > > gendered by
> > some
> > > > other criteria (e.g. reader interest, romantic comedy vs action
> film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l [mailto:
> > [hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of
> > > > Wikipedia citations and no one I've asked knows of any work on
> > > > this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings.
> > > > There
> is
> > > > also an initiative to place citations in an annotated structured
> > repository
> > > > (wikicite). I do not know the current status of wikicite or
> > > > if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a
> > > > sensible
> > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is  
> > > > necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community
> > > > is interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Thu, 22 Aug 2019 13:36:17 -0700
> > From: Leila Zia <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>
> > Subject: [Wiki-research-l] Wikimania 2019 disinformation meetup
> >         follow-up
> > Message-ID:
> >         <CAK0Oe2sodYJpkuhSqgo3dtfDr=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi,
> >
> > This message is for those of you who attended the disinformation
> > meet-up [0] in Wikimania 2019 [1] or others who may be interested.
> >
> > * The notes from our meet-up are now posted in the bottom of the
> > page
> [0].
> >
> > * I was tasked to see if space.wmflabs.org is the place for us to
> > continue conversations about this topic. The answer is yes. Thanks
> > to the help of Elena Lappen, we now have a dedicated subcategory for
> > disinformation:
> > https://discuss-space.wmflabs.org/c/research/disinformation . Feel
> > free to subscribe, watch, and/or post new topics if you're involved
> > in this space.
> >
> > * If you are new to this conversation, please read the purpose of
> > the subcategory at
> >
> https://discuss-space.wmflabs.org/t/about-the-disinformation-category/
> 949
> > and welcome! :)
> >
> > Best,
> > Leila
> >
> > [0] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation
> > [1] https://wikimania.wikimedia.org/wiki/2019:Program
> >
> >
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Thu, 22 Aug 2019 22:43:53 +0000 (UTC)
> > From: Mohammed Sadat Abdulai <[hidden email]>
> > To: Research Into Wikimedia Content and Communities
> >         <[hidden email]>
> > Subject: [Wiki-research-l] Upcoming Research Newsletter (special issue
> >         on gender gap research): New papers open for review
> > Message-ID: <[hidden email]>
> > Content-Type: text/plain; charset=UTF-8
> >
> >  Hi everyone,
> > We’re preparing for the August 2019 research newsletter and looking
> > for contributors. Please take a look at
> > https://etherpad.wikimedia.org/p/WRN201908 and add your name next to
> > any paper you are interested in covering. Our target publication
> > date is on
> 31
> > August 11:59 UTC. As usual, short notes and one-paragraph reviews
> > are
> most
> > welcome.
> >  For the August edition, we are planning a special issue focusing
> > mainly on recent gender gap/gender bias research. (Upcoming special
> > issues
> topics
> > may include health and education.) There are about 20 papers from
> > this
> area
> > on our todo list which will all be covered in the August issue,
> > either
> as a
> > mere list item or - with your help - in form of a more informative
> writeup
> > or review. They include:
> >    - Analyzing Gender Stereotyping in Bollywood Movies
> >
> >    - Breaking the glass ceiling on Wikipedia| journal
> >
> >    - Breastfeeding, Authority, and Genre: Women's Ethos in Wikipedia
> > and Blogs
> >
> >    - Cyberfeminism on Wikipedia: Visibility and deliberation in
> > feminist Wikiprojects
> >
> >    - Gender and deletion on Wikipedia
> >
> >    - Gender imbalance and Wikipedia
> >
> >    - Gender Markers in Wikipedia Usernames
> >
> >    - How do students trust Wikipedia? An examination across genders
> >
> >    - Investigating the Gender Pronoun Gap in Wikipedia
> >
> >    - It’s Not What You Think: Gender Bias in Information about
> > Fortune
> > 1000 CEOs on Wikipedia
> >
> >    - Mapping and Bridging the Gender Gap: An Ethnographic Study of
> > Indian Wikipedians and Their Motivations to Contribute
> >
> >    - People Who Can Take It: How Women Wikipedians Negotiate and
> > Navigate Safety
> >
> >    - Redressing Gender Inequities on Wikipedia Through an Editathon
> >
> >    - Similar Gaps, Different Origins? Women Readers and Editors at
> > Greek Wikipedia
> >
> >    - Simulation Experiments on (the Absence of) Ratings Bias in
> Reputation
> > Systems
> >
> >    - The Gendered Presentation of Professions on Wikipedia
> >
> >    - Who Counts as a Notable Sociologist on Wikipedia? Gender, Race,
> > and the “Professor Test”
> >
> >    - Who Wants to Read This?: A Method for Measuring Topical
> > Representativeness in User Generated Content Systems
> >
> >    - Women and Wikipedia. Diversifying Editors and Enhancing Content
> > through Library Edit-a-Thons
> >
> > Masssly and Tilman Bayer
> >
> > [1] Research:Newsletter - Meta[2] WikiResearch (@WikiResearch) on
> > Twitter
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 12
> > ************************************************
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 23 Aug 2019 14:41:09 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID: <001001d5596c$fe22a100$fa67e300$@gmail.com>
> Content-Type: text/plain;       charset="utf-8"
>
> Yes, that was my thought. It would be difficult to know the sex (or
> the
> gender) of an author name on a paper. There would inevitably be a lot
> that you could not determine. And certainly in the sciences
> multi-author pages are the norm and even where you did know the
> sex/gender of all, do you assign some part-score? E.g. 0 for all male,
> 1 for all female, 0.6 for 3 women and 2 men.
>
> But I am curious why you are asking the question? That the
> writing/research of women is under-represented in Wikipedia citations?
> If so, without conducting any research, I'd say "yes it is under-represented".
> But my reason would be because women are under-represented as
> writers/researchers in the first place.  And certainly the older the
> source, the more likely it is to be written by a man. So to
> investigate gender bias in citations in Wikipedia, you would have to
> estimate the proportion of men/women (or at least their outputs) over
> time in a given discipline and then ask the question, "taking into
> account of the time of publication of a citation and the proportion of
> men/women active in this discipline at that time, do Wikipedia citations show a sex/gender basis?".
> Hmm ... very tricky.
>
> I'd be inclined to suggest starting with a much simpler task. Pick a
> discipline (preferably one with a professional society who can tell
> your their estimate of current male/female ratio over (say) the past 5
> years), limit the Wikipedia articles to topics in that discipline, and
> limit the citations to those published within the last 5 years.
> Indeed, perhaps limiting it to publications that are principally from
> the same country(s) as the professional society from which you get the
> data (as clearly men/women's participation in any discipline can vary
> with different countries for cultural reasons). Then you have some way
> to gauge whether Wikipedia is showing more or less gender bias in its
> citations than the discipline itself exhibits through publication. Quite a challenge!
>
> And of course, it is not Wikipedia that adds citations. It is
> individual contributor who add citations. Does the sex/gender of the
> contributor have any correlation to any observed bias? Again, the task
> is made more difficult because a lot of Wikipedians don't identify their sex/gender.
>
> The other thing to be alert to is the difference in how (I believe)
> Wikipedians cite compared to researchers. As a researcher, I will of
> course be reading papers in my field all the time and what I read will
> influence my subsequent work. Therefore when I write about my
> research, my citations are referring to papers that I have already
> read and whose authors may be familiar to me from their other work,
> having met them at a conferences, private correspondence, etc. However
> as a Wikipedian, I am only partially operating that way (mostly when I
> write new articles or significantly expand them, that is, when I am
> doing the research). A lot of the time I am adding citations relating
> to content other people (often new users) have added/changed without citation. These come up on my watchlist all the time.
> What do I do? Of course I could revert saying "no citation provided",
> but that's not the way to encourage new contributors nor to grow the
> encyclopedia, so if the information seems plausible (not obviously
> vandalism), I will attempt to find a citation for it (using tools like
> Google and other topic-specialise search tools). This is what I call
> "lucky dip" mode of citing as obviously I have no idea what the source
> was for the original contributor. The sources I find from my search
> may not already be known to me (frequently they are not). Or to
> summarise, IMHO, researchers (or Wikipedians in "new content mode")
> cite a source already known to them and whose authors may be known to
> them and could consciously or unconsciously engage in some
> discrimination in citation based on sex/gender or other criteria,
> whereas Wikipedians in "updating mode" are likely to be citing a
> source not previously known to them and may be happy just to have
> found a source and are unlikely to be spending a lot of their time
> researching the authors of that source to be extent they could then
> consciously or unconsciously exercise discrimination on sex/gender. If
> I invest any extra effort in such a situations, it's probably because
> the wording of the source is a close match to the Wikipedia article
> which begs the question of copyright violation (which needs to be
> dealt with by deletion or rewriting) or being a Wikipedia mirror (which is obviously not an acceptable citation).
>
> So I suspect whether a citation was added by the same contributor as
> the content it supports or a subsequent contributor probably makes a
> difference to the likelihood of conscious/unconscious discrimination.
>
> Also, finally, often Wikipedia cites web pages and other sources that
> do not have any individual authorship, e.g. government websites.
> Remember that Wikipedia prefers open citations over paywalled
> citations and a lot of the publications behind paywalls are individually authored.
>
> Your proposed research has a lot of interesting challenges and a
> number of limitations. I'm not saying don't do it, but I am saying
> start very small and see if you can find any evidence to support your
> hypothesis before embarking on a larger study. Because contributor
> behaviour is what you are trying to study, you probably need to do
> both quantitative and qualitative experiments. E.g. I have described
> the two modes of citation I do, but I cannot say how typical my behaviour is.
>
> Kerry
>
> -----Original Message-----
> From: Wiki-research-l
> [mailto:[hidden email]]
> On Behalf Of Leila Zia
> Sent: Friday, 23 August 2019 3:44 AM
> To: Research into Wikimedia content and communities <
> [hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
>
> Hi Greg,
>
> A few comments if you're going to go with "proportion of male vs
> female authors of the source material used as citations in arbitrary
> articles":
>
> * Please differentiate between sex (female, male, ...) and gender
> (woman, man, ...). My understanding from your initial email is that
> you want to stay focused on gender, not sex.
>
> * Unless you have reliable sources about the gender of an author, I
> would not recommend trying to predict what the gender is. (As you may
> know, this is not uncommon in social media studies, for example, to
> predict the gender of the author based on their image or their name.
> These approaches introduce biases and social challenges.)
>
> * Re your question about whether WMF has resources to look into this
> question in-house: I can't speak for the whole of WMF, however, I can
> share more about the Research team's direction. As part of our future
> work, we would like to "help contributors monitor violations of core
> content policies and assess information reliability and bias both
> granularly and at scale". [1] The question you proposed can fall under
> assessing bias in content (considering citations as part of the
> content). I expect us to focus first on the piece about violations of
> core content policies and information reliability and come back to the
> bias question later. As a result, we won't have bandwidth to do your proposal in-house at the moment.
> Sorry about that.
>
> I hope this helps.
>
> Best,
> Leila
>
> [1] Section 2 of our Knowledge Integrity whitepaper:
>
> https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrit
> y_-_Wikimedia_Research_2030.pdf
>
>
> On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> >
> > Hi Kerry,
> > Those are all very interesting ways to look at this. I was thinking
> > mostly along the lines of your first bullet point, but I'd be
> > interested in research in any of those areas.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 5:00 AM
> > <[hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >        
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more
> > > specific than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > --
> > >
> > > Message: 1
> > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <
> > > [hidden email]
> > > CAOO9DNtY+>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of
> > > Wikipedia citations and no one I've asked knows of any work on
> > > this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings.
> > > There is also an initiative to place citations in an annotated
> > > structured repository (wikicite). I do not know the current status
> > > of wikicite or if/when this could be used for this inquiry--either
> > > to examine all, or a sensible subset of the citations.
> > >
> > > My perspective is that understanding the gender balance is
> > > necessary and urgent. The balance could be better, the same, or
> > > worse than the citation balances we already know, and the scale of
> > > the
> effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> > > Does the WMF have the resources and interest to look into this
> > > matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > From: "Kerry Raymond" <[hidden email]>
> > > To: "'Research into Wikimedia content and communities'"
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> > > citations
> > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > Content-Type: text/plain;       charset="UTF-8"
> > >
> > > Could you elaborate a bit more on what you mean by the gender
> > > balance of citations?
> > >
> > > Are you talking about:
> > >
> > > * proportion of male vs female authors of the source material used
> > > as citations in arbitrary articles>
> > > *  the quality/quantity of citations in biography articles of men
> > > vs
> women?
> > > * the quality/quantity of citations in articles that are gendered
> > > by some other criteria (e.g. reader interest, romantic comedy vs
> > > action
> film)?
> > >
> > > Kerry
> > >
> > > -----Original Message-----
> > > From: Wiki-research-l
> > > [mailto:[hidden email]]
> > > On Behalf Of Greg
> > > Sent: Thursday, 22 August 2019 1:19 PM
> > > To: [hidden email]
> > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > >
> > > Greetings!
> > >
> > > I was looking for information about the gender balance of
> > > Wikipedia citations and no one I've asked knows of any work on
> > > this topic. Do
> you?
> > >
> > > I think this is an important question.
> > >
> > > Here's what I've learned so far:
> > >
> > > Wikipedia citations are currently in the form of text strings.
> > > There is also an initiative to place citations in an annotated
> > > structured repository (wikicite). I do not know the current status
> > > of wikicite or if/when this could be used for this inquiry--either
> > > to examine all, or a sensible subset of the citations.
> > >
> > > My perspective is that understanding the gender balance is
> > > necessary and urgent. The balance could be better, the same, or
> > > worse than the citation balances we already know, and the scale of
> > > the
> effect is quite large.
> > >
> > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > interested in pursuing? If so, what is the best way to get started?
> > > Does the WMF have the resources and interest to look into this
> > > matter
> inhouse?
> > >
> > > Thanks for your thoughts.
> > >
> > > Greg
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > ************************************************
> > >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 13
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Greg-2
In reply to this post by Greg-2
Thanks again, Kerry. I am hoping that someone with access to more resources
(knowledge, support, etc) than I have will look into this.

A few more thoughts/questions:

1. The link to the citation dataset from the Medium article ("What are the
ten most cited sources on Wikipedia? Let’s ask the data.") is broken.
2. As far as I can tell, every named author in the top ten most cited
sources on Wikipedia is male. One piece is by a working group
3. This line from the Medium piece struck me: "Many of these publications
have been cited by Wikipedians across large series of articles using
powerful bots and automated tools."

Are citations being added by bots? I'm not sure that I understand that line
correctly.

Greg


On Sat, Aug 24, 2019 at 1:51 AM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. Re: sockpuppets and how to find them sooner (Timothy Wood)
>    2. Re: gender balance of Wikipedia citations (Kerry Raymond)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 23 Aug 2019 17:51:41 -0400
> From: Timothy Wood <[hidden email]>
> To: Kerry Raymond <[hidden email]>
> Cc: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID:
>         <CAMy3BEJ8=E1FgifdqgY+=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Then again, apparently the Foundation has a PR team whose only job is to
> compile the latest marketing buzzwords, and they seem to really love AI.
> You might get some buy in. Never know.
>
> V/r
> TJW/GMG
>
> On Fri, Aug 23, 2019, 11:23 Kerry Raymond <[hidden email]> wrote:
>
> > That's why I think we need "signatures" which is my shorthand for things
> > like a hash function or a bounding box, a means by which many
> non-matching
> > accounts can be eliminated at low cost, reserving the high cost
> comparisons
> > (machine or human) only for high probability candidates. It is
> > machine-computed and *stored* on the banning/blocking of a user. When a
> > suspect user is presented, it calculates their signature and then
> compares
> > them against the pre-calculated signatures of the bad users. I don't
> think
> > it is too expensive if we can find the right "signature". CPU cycles are
> > pretty fast. I only have an average laptop CPU-wise but I burn through
> > loads of comparisons of geographic boundaries (complex polygons with many
> > points) thanks to bounding boxes which reduce the complex shape to the
> > smallest rectangle that contains it. Testing intersection of polygons is
> > expensive, testing the intersection of rectangles is trivial.
> >
> > I think we can probably ignore the myriad of trivial bad guys for the
> > purposes of signature collecting, eg blocked for vandalism after their
> > first few edits. Sock puppets or their masters don't immediately appear
> as
> > bad guys on individual edits. It's often more about long-term behaviours
> > like POV pushing, refusal to engage in consensus building, slow burning
> > edit wars, etc, that does not show on individual edits.
> >
> > Kerry
> >
> > Sent from my iPad
> >
> > On 23 Aug 2019, at 11:42 pm, Timothy Wood <[hidden email]>
> > wrote:
> >
> > You are correct that in all but the most obvious cases, filing an SPI can
> > be exceptionally time consuming. I'm afraid there is no obvious technical
> > solution there that would not involve a complicated AI that is probably
> > beyond the ability of the foundation to produce.
> >
> > There is quite a bit of data available in the form of years of SPIs, but
> > it seems like you're talking about Facebook or Google levels of machine
> > learning, and even years of SPIs is tiny compared to the amount of data
> > they work with.
> >
> > On a separate note, frequently changing IP adresses is most often an
> > indicator of nothing more than someone who is editing on a mobile
> > connection. This can usually be easily verified with an online IP lookup.
> >
> > V/r
> > TJW/GMG
> >
> >
> >
> > On Fri, Aug 23, 2019, 02:44 RhinosF1 <[hidden email]> wrote:
> >
> >> Just a note that you can still go through warnings for vandalism etc.
> and
> >> report to AIV.
> >>
> >> Or at that edit speed, you may have a chance at AN at reporting for
> >> bot-like edits which will draw attention to the account.
> >>
> >> If you ever need help, things like #wikipedia-en-help on Freenode IRC
> >> exist
> >> so you can ask other users.
> >>
> >> RhinosF1
> >> Miraheze Volunteer
> >>
> >> On Fri, 23 Aug 2019 at 06:57, Kerry Raymond <[hidden email]>
> >> wrote:
> >>
> >> > Currently, to open a sockpuppet investigation, you must name the two
> (or
> >> > more) accounts that you believe to be sockpuppets with "clear,
> >> behavioural
> >> > evidence of sock puppetry" which is typically in the form of pairs of
> >> edits
> >> > that demonstrate similar edit behaviours that are unlikely to
> naturally
> >> > occur. Now if you spend enough time on-wiki, you develop an intuition
> >> about
> >> > behaviours you see on your watchlist and in article edit histories.
> >> Often I
> >> > am highly suspicious that an account is a sockpuppet, but I cannot
> >> report
> >> > them because I don't know which other account is involved.
> >> >
> >> >
> >> >
> >> > As a example, I recently encounted User:Shelati an account about 1 day
> >> old
> >> > at that time with nearly 100 edits in that day all about 1-2 minutes
> >> apart,
> >> > mostly making a similar change to a large number of Australian place
> >> > infoboxes.
> >> >
> >> >
> >> >
> >> >
> >>
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati
> >> > <
> >> >
> >>
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&of
> >> > fset=20190728053057&limit=100&target=Shelati
> >> > <
> >>
> https://en.wikipedia.org/w/index.php?title=Special:Contributions/Shelati&offset=20190728053057&limit=100&target=Shelati
> >> >
> >> > >
> >> > &offset=20190728053057&limit=100&target=Shelati
> >> >
> >> >
> >> >
> >> > Genuine new users do not edit that quickly, do not use templates and
> do
> >> not
> >> > mess structurally with infoboxes (at most they try to change the
> >> values).
> >> > It
> >> > "smelled" like a sockpuppet. However, as I did not recognise that
> >> pattern
> >> > of
> >> > edit behaviour as being that of any other user I was familiar with, it
> >> > wasn't something I could report for sockpuppet investigation. Anyhow
> >> after
> >> > about 2 weeks, the user was blocked as a sockpuppet. Someone must have
> >> > noticed and figured out the other account:
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> https://en.wikipedia.org/wiki/Wikipedia:Sockpuppet_investigations/Meganesia/
> >> > Archive
> >> >
> >> >
> >> >
> >> > Two weeks and 1,279 edits later . that's over 1000 possibly
> problematic
> >> > edits after I first suspected them. But that's nothing compared with
> >> > another
> >> > ongoing situation in which a very large number of different IPs are
> >> engaged
> >> > in a pattern of problem edits on mostly Australian articles (a few
> >> > different
> >> > types of edits but an obvious "quack like a duck" situation). The IP
> >> number
> >> > changes frequently (and one assumes deliberately). The edits
> >> potentially go
> >> > back to 2013 but appear to have intensified in 2018/2019. Here's one
> >> user's
> >> > summary of all the IP addresses involved, and the extent to which they
> >> have
> >> > been cleaned up, given many thousands of edits are involved, see:
> >> >
> >> >
> >> >
> >> > https://en.wikipedia.org/wiki/User:IamNotU/History_cleanup
> >> >
> >> >
> >> >
> >> > As well as the damage done to the content (which harms the readers),
> >> these
> >> > IP sockpuppets are consuming enormous amounts of effort to track them
> >> down
> >> > and revert them, which could be more productively used to improve the
> >> > content. We need better tools to foil these pests. So I want to put
> that
> >> > challenge out to this list.
> >> >
> >> >
> >> >
> >> > Kerry
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Wiki-research-l mailing list
> >> > [hidden email]
> >> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >> >
> >> --
> >> RhinosF1
> >> Miraheze Volunteer
> >> _______________________________________________
> >> Wiki-research-l mailing list
> >> [hidden email]
> >> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >>
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Sat, 24 Aug 2019 18:29:36 +1000
> From: "Kerry Raymond" <[hidden email]>
> To: "'Research into Wikimedia content and communities'"
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> Message-ID: <004801d55a56$11f4a270$35dde750$@gmail.com>
> Content-Type: text/plain;       charset="utf-8"
>
> I am inclined to think that political science has more Point of View in it
> than say chemistry. I also suspect it has fewer authors per book/paper. So
> I can imagine that people citing political science literature may be more
> inclined to cherry pick the sources that support their own POV which may
> involve some gender bias in some way. I would think it less likely in
> chemistry to cherry pick sources (which is not to say there are no divided
> schools of thought in chemistry but it is a more experimental discipline
> with strong commitment to factual data and less to opinion).
>
> But having said all that, whether and in what circumstances that the
> selection of sources in Wikipedia might be sex/gender biased, I honestly
> don't know. But if it manifests outside of Wikipedia as you suggest, then I
> would be very surprised if it wasn't replicated in Wikipedia to some
> extent. But I guess your question is whether is Wikipedia merely reflects
> the society it lives in (similar levels of gender bias) or whether there is
> something about Wikipedia which acerbates or ameliorates the situation? I
> am genuine curious what a small study would discover and agree that
> replicating (as much as possible) the existing study outside of Wikipedia)
> provides a good starting point. You might approach the authors of that
> study to see if they are willing to collaborate on such a project, either
> in design, data sharing or more fully. I look forward to seeing the results.
>
> Kerry
>
> -----Original Message-----
> From: Wiki-research-l [mailto:[hidden email]]
> On Behalf Of Greg
> Sent: Friday, 23 August 2019 5:01 PM
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
>
> Wow, Kerry! Thank you for taking the time to write all these thoughts out.
>
> I'm asking the question because I'm concerned that the gender balance of
> the authors being cited on wikipedia is different from the already quite
> bad patterns in academia. My fear is that the citation gender imbalance on
> Wikipedia is more pronounced. If so, it is not just perpetuating the
> problem, but making it worse by surfacing certain authors and ideas even
> more frequently, or hardly at all. I would like to know if this is the
> case, and if so, how big the effect is.
>
> In my last message, I mention a study about a set of award-winning
> political science books (the researchers study the citation gender
> imbalance for that set). I just saw this study today, but I began to think
> that it/the set of works--or some similar set of titles--could possibly be
> a good place to begin, especially if the original researchers were willing
> to share the list of titles/authors/gender/etc that they put
> together/worked with. Then it seems it would mostly be a matter of figuring
> out how to understand how those titles are cited on Wikipedia--through
> either the citation dataset or wikicite--to see if/how the citation
> patterns differ (i.e., if the works by women/men are cited more
> frequently/at the same rate/less frequently on Wikipedia than what the
> researchers found in the original study).
>
> This seems like it would be easier to do than what you propose, but
> perhaps the idea is not sound. Until very recently, I thought I could find
> the answer in an existing paper! I honestly don't know the best way to get
> the answer, but I would like to know the answer and think it's important to
> look at.
>
> All of the things you bring up--from the gender of the editor, to the type
> of editing being done, to the issues around multiple authors/paywalls/year
> of publication/field--complicate the inquiry, and in particular a larger
> one. I agree with what you say about doing something small first to see
> what's there.
>
> Thanks again for all your thoughts.
> Greg
>
>
>
> On Thu, Aug 22, 2019 at 9:41 PM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: gender balance of wikipedia citations (Greg)
> >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 22 Aug 2019 18:47:48 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID:
> >         <
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Leila,
> >
> > Thanks for your thoughts.
> >
> > Having just read Troy Vettese's very powerful essay, Sexism in the
> > Academy (
> > https://nplusonemag.com/issue-34/essays/sexism-in-the-academy/), I
> > wish this were a top priority.
> >
> > I stumbled upon a study today--it came up in the Washington Post's
> > excellent series on gender bias in political science. The authors look
> > at a set of award winning political science books and the gender
> > imbalance in the citations drawn from google scholar.  I'm linking the
> > piece here in case anyone on this list is interested now, or in the
> > future, in how the patterns on Wikipedia compare.
> >
> > Washington Post piece: "There’s a gender gap in who wins political
> > science book awards – and in how widely they’re cited"
> >
> > https://www.washingtonpost.com/politics/2019/08/22/theres-gender-gap-w
> > ho-wins-political-science-book-awards-how-widely-theyre-cited/
> > "Just as significantly, women’s award-winning books receive fewer
> > scholarly citations than men’s award-winning volumes — and this
> > disparity has grown, rather than shrunk, in recent years. Over the
> > entire period, APSA award-winning volumes by women averaged 43 percent
> > fewer citations per year than those by male authors."
> >
> > Paper: "Winning awards and gaining recognition: An impact analysis of
> > APSA section book prizes"
> > https://www.sciencedirect.com/science/article/abs/pii/S036233191830086
> > 7
> >
> >
> > Best,
> > Greg
> >
> > On Thu, Aug 22, 2019 at 3:44 PM <
> > [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. Re: gender balance of wikipedia citations (Greg)
> > >    2. Re: gender balance of wikipedia citations (Leila Zia)
> > >    3. Wikimania 2019 disinformation meetup follow-up (Leila Zia)
> > >    4. Upcoming Research Newsletter (special issue on gender gap
> > >       research): New papers open for review (Mohammed Sadat Abdulai)
> > >
> > >
> > > --------------------------------------------------------------------
> > > --
> > >
> > > Message: 1
> > > Date: Thu, 22 Aug 2019 09:57:15 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <CAOO9DNuSYzzaVwcdqiWA7pj671z3N43XOSwv6DtW0SxWg=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > mostly
> > > along the lines of your first bullet point, but I'd be interested in
> > > research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more
> > > > specific than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > ----
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]
> > > > CAOO9DNtY+>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of
> > > > Wikipedia citations and no one I've asked knows of any work on
> > > > this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings.
> > > > There is also an initiative to place citations in an annotated
> > > > structured
> > > repository
> > > > (wikicite). I do not know the current status of wikicite or
> > > > if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a
> > > > sensible
> > > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> > > > citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender
> > > > balance
> > of
> > > > citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used
> > > > as citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men
> > > > vs
> > > women?
> > > > * the quality/quantity of citations in articles that are gendered
> > > > by
> > some
> > > > other criteria (e.g. reader interest, romantic comedy vs action
> film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l [mailto:
> > > [hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of
> > > > Wikipedia citations and no one I've asked knows of any work on
> > > > this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings.
> > > > There is also an initiative to place citations in an annotated
> > > > structured
> > > repository
> > > > (wikicite). I do not know the current status of wikicite or
> > > > if/when
> > this
> > > > could be used for this inquiry--either to examine all, or a
> > > > sensible
> > > subset
> > > > of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary
> > and
> > > > urgent. The balance could be better, the same, or worse than the
> > citation
> > > > balances we already know, and the scale of the effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > Does
> > > > the WMF have the resources and interest to look into this matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 22 Aug 2019 10:43:51 -0700
> > > From: Leila Zia <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID:
> > >         <CAK0Oe2uCo70_=ma2b=2d+fvr4GseEVxOP0sh=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Greg,
> > >
> > > A few comments if you're going to go with "proportion of male vs
> > > female authors of the source material used as citations in arbitrary
> > > articles":
> > >
> > > * Please differentiate between sex (female, male, ...) and gender
> > > (woman, man, ...). My understanding from your initial email is that
> > > you want to stay focused on gender, not sex.
> > >
> > > * Unless you have reliable sources about the gender of an author, I
> > > would not recommend trying to predict what the gender is. (As you
> > > may know, this is not uncommon in social media studies, for example,
> > > to predict the gender of the author based on their image or their name.
> > > These approaches introduce biases and social challenges.)
> > >
> > > * Re your question about whether WMF has resources to look into this
> > > question in-house: I can't speak for the whole of WMF, however, I
> > > can share more about the Research team's direction. As part of our
> > > future work, we would like to "help contributors monitor violations
> > > of core content policies and assess information reliability and bias
> > > both granularly and at scale". [1] The question you proposed can
> > > fall under assessing bias in content (considering citations as part
> > > of the content). I expect us to focus first on the piece about
> > > violations of core content policies and information reliability and
> > > come back to the bias question later. As a result, we won't have
> > > bandwidth to do your proposal in-house at the moment. Sorry about that.
> > >
> > > I hope this helps.
> > >
> > > Best,
> > > Leila
> > >
> > > [1] Section 2 of our Knowledge Integrity whitepaper:
> > >
> > >
> > https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrit
> > y_-_Wikimedia_Research_2030.pdf
> > >
> > >
> > > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]>
> wrote:
> > > >
> > > > Hi Kerry,
> > > > Those are all very interesting ways to look at this. I was
> > > > thinking
> > > mostly
> > > > along the lines of your first bullet point, but I'd be interested
> > > > in research in any of those areas.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Thu, Aug 22, 2019 at 5:00 AM <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Send Wiki-research-l mailing list submissions to
> > > > >         [hidden email]
> > > > >
> > > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > > >
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > > or, via email, send a message with subject or body 'help' to
> > > > >         [hidden email]
> > > > >
> > > > > You can reach the person managing the list at
> > > > >         [hidden email]
> > > > >
> > > > > When replying, please edit your Subject line so it is more
> > > > > specific than "Re: Contents of Wiki-research-l digest..."
> > > > >
> > > > >
> > > > > Today's Topics:
> > > > >
> > > > >    1. gender balance of wikipedia citations (Greg)
> > > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > > >
> > > > >
> > > > >
> > ----------------------------------------------------------------------
> > > > >
> > > > > Message: 1
> > > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > > From: Greg <[hidden email]>
> > > > > To: [hidden email]
> > > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > > Message-ID:
> > > > >         <
> > > > > [hidden email]
> > > > > CAOO9DNtY+om>
> > > > > Content-Type: text/plain; charset="UTF-8"
> > > > >
> > > > > Greetings!
> > > > >
> > > > > I was looking for information about the gender balance of
> > > > > Wikipedia citations and no one I've asked knows of any work on
> > > > > this topic. Do
> > > you?
> > > > >
> > > > > I think this is an important question.
> > > > >
> > > > > Here's what I've learned so far:
> > > > >
> > > > > Wikipedia citations are currently in the form of text strings.
> > > > > There
> > is
> > > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > > (wikicite). I do not know the current status of wikicite or
> > > > > if/when
> > > this
> > > > > could be used for this inquiry--either to examine all, or a
> > > > > sensible
> > > subset
> > > > > of the citations.
> > > > >
> > > > > My perspective is that understanding the gender balance is
> > > > > necessary
> > > and
> > > > > urgent. The balance could be better, the same, or worse than the
> > > citation
> > > > > balances we already know, and the scale of the effect is quite
> large.
> > > > >
> > > > > Is this a line of inquiry that the wikimedia/wikicite community
> > > > > is interested in pursuing? If so, what is the best way to get
> started?
> > > Does
> > > > > the WMF have the resources and interest to look into this matter
> > > inhouse?
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > Greg
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > Message: 2
> > > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > > From: "Kerry Raymond" <[hidden email]>
> > > > > To: "'Research into Wikimedia content and communities'"
> > > > >         <[hidden email]>
> > > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> > > > > citations
> > > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > > Content-Type: text/plain;       charset="UTF-8"
> > > > >
> > > > > Could you elaborate a bit more on what you mean by the gender
> > > > > balance
> > > of
> > > > > citations?
> > > > >
> > > > > Are you talking about:
> > > > >
> > > > > * proportion of male vs female authors of the source material
> > > > > used as citations in arbitrary articles>
> > > > > *  the quality/quantity of citations in biography articles of
> > > > > men vs
> > > women?
> > > > > * the quality/quantity of citations in articles that are
> > > > > gendered by
> > > some
> > > > > other criteria (e.g. reader interest, romantic comedy vs action
> > film)?
> > > > >
> > > > > Kerry
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wiki-research-l [mailto:
> > > [hidden email]]
> > > > > On Behalf Of Greg
> > > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > > To: [hidden email]
> > > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > >
> > > > > Greetings!
> > > > >
> > > > > I was looking for information about the gender balance of
> > > > > Wikipedia citations and no one I've asked knows of any work on
> > > > > this topic. Do
> > > you?
> > > > >
> > > > > I think this is an important question.
> > > > >
> > > > > Here's what I've learned so far:
> > > > >
> > > > > Wikipedia citations are currently in the form of text strings.
> > > > > There
> > is
> > > > > also an initiative to place citations in an annotated structured
> > > repository
> > > > > (wikicite). I do not know the current status of wikicite or
> > > > > if/when
> > > this
> > > > > could be used for this inquiry--either to examine all, or a
> > > > > sensible
> > > subset
> > > > > of the citations.
> > > > >
> > > > > My perspective is that understanding the gender balance is
> > > > > necessary
> > > and
> > > > > urgent. The balance could be better, the same, or worse than the
> > > citation
> > > > > balances we already know, and the scale of the effect is quite
> large.
> > > > >
> > > > > Is this a line of inquiry that the wikimedia/wikicite community
> > > > > is interested in pursuing? If so, what is the best way to get
> started?
> > > Does
> > > > > the WMF have the resources and interest to look into this matter
> > > inhouse?
> > > > >
> > > > > Thanks for your thoughts.
> > > > >
> > > > > Greg
> > > > > _______________________________________________
> > > > > Wiki-research-l mailing list
> > > > > [hidden email]
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > Subject: Digest Footer
> > > > >
> > > > > _______________________________________________
> > > > > Wiki-research-l mailing list
> > > > > [hidden email]
> > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > >
> > > > >
> > > > > ------------------------------
> > > > >
> > > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > > ************************************************
> > > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 3
> > > Date: Thu, 22 Aug 2019 13:36:17 -0700
> > > From: Leila Zia <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: [Wiki-research-l] Wikimania 2019 disinformation meetup
> > >         follow-up
> > > Message-ID:
> > >         <CAK0Oe2sodYJpkuhSqgo3dtfDr=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi,
> > >
> > > This message is for those of you who attended the disinformation
> > > meet-up [0] in Wikimania 2019 [1] or others who may be interested.
> > >
> > > * The notes from our meet-up are now posted in the bottom of the
> > > page
> > [0].
> > >
> > > * I was tasked to see if space.wmflabs.org is the place for us to
> > > continue conversations about this topic. The answer is yes. Thanks
> > > to the help of Elena Lappen, we now have a dedicated subcategory for
> > > disinformation:
> > > https://discuss-space.wmflabs.org/c/research/disinformation . Feel
> > > free to subscribe, watch, and/or post new topics if you're involved
> > > in this space.
> > >
> > > * If you are new to this conversation, please read the purpose of
> > > the subcategory at
> > >
> > https://discuss-space.wmflabs.org/t/about-the-disinformation-category/
> > 949
> > > and welcome! :)
> > >
> > > Best,
> > > Leila
> > >
> > > [0] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation
> > > [1] https://wikimania.wikimedia.org/wiki/2019:Program
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 4
> > > Date: Thu, 22 Aug 2019 22:43:53 +0000 (UTC)
> > > From: Mohammed Sadat Abdulai <[hidden email]>
> > > To: Research Into Wikimedia Content and Communities
> > >         <[hidden email]>
> > > Subject: [Wiki-research-l] Upcoming Research Newsletter (special issue
> > >         on gender gap research): New papers open for review
> > > Message-ID: <[hidden email]>
> > > Content-Type: text/plain; charset=UTF-8
> > >
> > >  Hi everyone,
> > > We’re preparing for the August 2019 research newsletter and looking
> > > for contributors. Please take a look at
> > > https://etherpad.wikimedia.org/p/WRN201908 and add your name next to
> > > any paper you are interested in covering. Our target publication
> > > date is on
> > 31
> > > August 11:59 UTC. As usual, short notes and one-paragraph reviews
> > > are
> > most
> > > welcome.
> > >  For the August edition, we are planning a special issue focusing
> > > mainly on recent gender gap/gender bias research. (Upcoming special
> > > issues
> > topics
> > > may include health and education.) There are about 20 papers from
> > > this
> > area
> > > on our todo list which will all be covered in the August issue,
> > > either
> > as a
> > > mere list item or - with your help - in form of a more informative
> > writeup
> > > or review. They include:
> > >    - Analyzing Gender Stereotyping in Bollywood Movies
> > >
> > >    - Breaking the glass ceiling on Wikipedia| journal
> > >
> > >    - Breastfeeding, Authority, and Genre: Women's Ethos in Wikipedia
> > > and Blogs
> > >
> > >    - Cyberfeminism on Wikipedia: Visibility and deliberation in
> > > feminist Wikiprojects
> > >
> > >    - Gender and deletion on Wikipedia
> > >
> > >    - Gender imbalance and Wikipedia
> > >
> > >    - Gender Markers in Wikipedia Usernames
> > >
> > >    - How do students trust Wikipedia? An examination across genders
> > >
> > >    - Investigating the Gender Pronoun Gap in Wikipedia
> > >
> > >    - It’s Not What You Think: Gender Bias in Information about
> > > Fortune
> > > 1000 CEOs on Wikipedia
> > >
> > >    - Mapping and Bridging the Gender Gap: An Ethnographic Study of
> > > Indian Wikipedians and Their Motivations to Contribute
> > >
> > >    - People Who Can Take It: How Women Wikipedians Negotiate and
> > > Navigate Safety
> > >
> > >    - Redressing Gender Inequities on Wikipedia Through an Editathon
> > >
> > >    - Similar Gaps, Different Origins? Women Readers and Editors at
> > > Greek Wikipedia
> > >
> > >    - Simulation Experiments on (the Absence of) Ratings Bias in
> > Reputation
> > > Systems
> > >
> > >    - The Gendered Presentation of Professions on Wikipedia
> > >
> > >    - Who Counts as a Notable Sociologist on Wikipedia? Gender, Race,
> > > and the “Professor Test”
> > >
> > >    - Who Wants to Read This?: A Method for Measuring Topical
> > > Representativeness in User Generated Content Systems
> > >
> > >    - Women and Wikipedia. Diversifying Editors and Enhancing Content
> > > through Library Edit-a-Thons
> > >
> > > Masssly and Tilman Bayer
> > >
> > > [1] Research:Newsletter - Meta[2] WikiResearch (@WikiResearch) on
> > > Twitter
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 12
> > > ************************************************
> > >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Fri, 23 Aug 2019 14:41:09 +1000
> > From: "Kerry Raymond" <[hidden email]>
> > To: "'Research into Wikimedia content and communities'"
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <001001d5596c$fe22a100$fa67e300$@gmail.com>
> > Content-Type: text/plain;       charset="utf-8"
> >
> > Yes, that was my thought. It would be difficult to know the sex (or
> > the
> > gender) of an author name on a paper. There would inevitably be a lot
> > that you could not determine. And certainly in the sciences
> > multi-author pages are the norm and even where you did know the
> > sex/gender of all, do you assign some part-score? E.g. 0 for all male,
> > 1 for all female, 0.6 for 3 women and 2 men.
> >
> > But I am curious why you are asking the question? That the
> > writing/research of women is under-represented in Wikipedia citations?
> > If so, without conducting any research, I'd say "yes it is
> under-represented".
> > But my reason would be because women are under-represented as
> > writers/researchers in the first place.  And certainly the older the
> > source, the more likely it is to be written by a man. So to
> > investigate gender bias in citations in Wikipedia, you would have to
> > estimate the proportion of men/women (or at least their outputs) over
> > time in a given discipline and then ask the question, "taking into
> > account of the time of publication of a citation and the proportion of
> > men/women active in this discipline at that time, do Wikipedia citations
> show a sex/gender basis?".
> > Hmm ... very tricky.
> >
> > I'd be inclined to suggest starting with a much simpler task. Pick a
> > discipline (preferably one with a professional society who can tell
> > your their estimate of current male/female ratio over (say) the past 5
> > years), limit the Wikipedia articles to topics in that discipline, and
> > limit the citations to those published within the last 5 years.
> > Indeed, perhaps limiting it to publications that are principally from
> > the same country(s) as the professional society from which you get the
> > data (as clearly men/women's participation in any discipline can vary
> > with different countries for cultural reasons). Then you have some way
> > to gauge whether Wikipedia is showing more or less gender bias in its
> > citations than the discipline itself exhibits through publication. Quite
> a challenge!
> >
> > And of course, it is not Wikipedia that adds citations. It is
> > individual contributor who add citations. Does the sex/gender of the
> > contributor have any correlation to any observed bias? Again, the task
> > is made more difficult because a lot of Wikipedians don't identify their
> sex/gender.
> >
> > The other thing to be alert to is the difference in how (I believe)
> > Wikipedians cite compared to researchers. As a researcher, I will of
> > course be reading papers in my field all the time and what I read will
> > influence my subsequent work. Therefore when I write about my
> > research, my citations are referring to papers that I have already
> > read and whose authors may be familiar to me from their other work,
> > having met them at a conferences, private correspondence, etc. However
> > as a Wikipedian, I am only partially operating that way (mostly when I
> > write new articles or significantly expand them, that is, when I am
> > doing the research). A lot of the time I am adding citations relating
> > to content other people (often new users) have added/changed without
> citation. These come up on my watchlist all the time.
> > What do I do? Of course I could revert saying "no citation provided",
> > but that's not the way to encourage new contributors nor to grow the
> > encyclopedia, so if the information seems plausible (not obviously
> > vandalism), I will attempt to find a citation for it (using tools like
> > Google and other topic-specialise search tools). This is what I call
> > "lucky dip" mode of citing as obviously I have no idea what the source
> > was for the original contributor. The sources I find from my search
> > may not already be known to me (frequently they are not). Or to
> > summarise, IMHO, researchers (or Wikipedians in "new content mode")
> > cite a source already known to them and whose authors may be known to
> > them and could consciously or unconsciously engage in some
> > discrimination in citation based on sex/gender or other criteria,
> > whereas Wikipedians in "updating mode" are likely to be citing a
> > source not previously known to them and may be happy just to have
> > found a source and are unlikely to be spending a lot of their time
> > researching the authors of that source to be extent they could then
> > consciously or unconsciously exercise discrimination on sex/gender. If
> > I invest any extra effort in such a situations, it's probably because
> > the wording of the source is a close match to the Wikipedia article
> > which begs the question of copyright violation (which needs to be
> > dealt with by deletion or rewriting) or being a Wikipedia mirror (which
> is obviously not an acceptable citation).
> >
> > So I suspect whether a citation was added by the same contributor as
> > the content it supports or a subsequent contributor probably makes a
> > difference to the likelihood of conscious/unconscious discrimination.
> >
> > Also, finally, often Wikipedia cites web pages and other sources that
> > do not have any individual authorship, e.g. government websites.
> > Remember that Wikipedia prefers open citations over paywalled
> > citations and a lot of the publications behind paywalls are individually
> authored.
> >
> > Your proposed research has a lot of interesting challenges and a
> > number of limitations. I'm not saying don't do it, but I am saying
> > start very small and see if you can find any evidence to support your
> > hypothesis before embarking on a larger study. Because contributor
> > behaviour is what you are trying to study, you probably need to do
> > both quantitative and qualitative experiments. E.g. I have described
> > the two modes of citation I do, but I cannot say how typical my
> behaviour is.
> >
> > Kerry
> >
> > -----Original Message-----
> > From: Wiki-research-l
> > [mailto:[hidden email]]
> > On Behalf Of Leila Zia
> > Sent: Friday, 23 August 2019 3:44 AM
> > To: Research into Wikimedia content and communities <
> > [hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> >
> > Hi Greg,
> >
> > A few comments if you're going to go with "proportion of male vs
> > female authors of the source material used as citations in arbitrary
> > articles":
> >
> > * Please differentiate between sex (female, male, ...) and gender
> > (woman, man, ...). My understanding from your initial email is that
> > you want to stay focused on gender, not sex.
> >
> > * Unless you have reliable sources about the gender of an author, I
> > would not recommend trying to predict what the gender is. (As you may
> > know, this is not uncommon in social media studies, for example, to
> > predict the gender of the author based on their image or their name.
> > These approaches introduce biases and social challenges.)
> >
> > * Re your question about whether WMF has resources to look into this
> > question in-house: I can't speak for the whole of WMF, however, I can
> > share more about the Research team's direction. As part of our future
> > work, we would like to "help contributors monitor violations of core
> > content policies and assess information reliability and bias both
> > granularly and at scale". [1] The question you proposed can fall under
> > assessing bias in content (considering citations as part of the
> > content). I expect us to focus first on the piece about violations of
> > core content policies and information reliability and come back to the
> > bias question later. As a result, we won't have bandwidth to do your
> proposal in-house at the moment.
> > Sorry about that.
> >
> > I hope this helps.
> >
> > Best,
> > Leila
> >
> > [1] Section 2 of our Knowledge Integrity whitepaper:
> >
> > https://upload.wikimedia.org/wikipedia/commons/9/9a/Knowledge_Integrit
> > y_-_Wikimedia_Research_2030.pdf
> >
> >
> > On Thu, Aug 22, 2019 at 9:57 AM Greg <[hidden email]> wrote:
> > >
> > > Hi Kerry,
> > > Those are all very interesting ways to look at this. I was thinking
> > > mostly along the lines of your first bullet point, but I'd be
> > > interested in research in any of those areas.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Aug 22, 2019 at 5:00 AM
> > > <[hidden email]>
> > > wrote:
> > >
> > > > Send Wiki-research-l mailing list submissions to
> > > >         [hidden email]
> > > >
> > > > To subscribe or unsubscribe via the World Wide Web, visit
> > > >
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > > or, via email, send a message with subject or body 'help' to
> > > >         [hidden email]
> > > >
> > > > You can reach the person managing the list at
> > > >         [hidden email]
> > > >
> > > > When replying, please edit your Subject line so it is more
> > > > specific than "Re: Contents of Wiki-research-l digest..."
> > > >
> > > >
> > > > Today's Topics:
> > > >
> > > >    1. gender balance of wikipedia citations (Greg)
> > > >    2. Re: gender balance of wikipedia citations (Kerry Raymond)
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > --
> > > >
> > > > Message: 1
> > > > Date: Wed, 21 Aug 2019 20:19:18 -0700
> > > > From: Greg <[hidden email]>
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > > Message-ID:
> > > >         <
> > > > [hidden email]
> > > > CAOO9DNtY+>
> > > > Content-Type: text/plain; charset="UTF-8"
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of
> > > > Wikipedia citations and no one I've asked knows of any work on
> > > > this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings.
> > > > There is also an initiative to place citations in an annotated
> > > > structured repository (wikicite). I do not know the current status
> > > > of wikicite or if/when this could be used for this inquiry--either
> > > > to examine all, or a sensible subset of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary and urgent. The balance could be better, the same, or
> > > > worse than the citation balances we already know, and the scale of
> > > > the
> > effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > > > Does the WMF have the resources and interest to look into this
> > > > matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Message: 2
> > > > Date: Thu, 22 Aug 2019 13:53:45 +1000
> > > > From: "Kerry Raymond" <[hidden email]>
> > > > To: "'Research into Wikimedia content and communities'"
> > > >         <[hidden email]>
> > > > Subject: Re: [Wiki-research-l] gender balance of wikipedia
> > > > citations
> > > > Message-ID: <00ed01d5589d$33e31ed0$9ba95c70$@gmail.com>
> > > > Content-Type: text/plain;       charset="UTF-8"
> > > >
> > > > Could you elaborate a bit more on what you mean by the gender
> > > > balance of citations?
> > > >
> > > > Are you talking about:
> > > >
> > > > * proportion of male vs female authors of the source material used
> > > > as citations in arbitrary articles>
> > > > *  the quality/quantity of citations in biography articles of men
> > > > vs
> > women?
> > > > * the quality/quantity of citations in articles that are gendered
> > > > by some other criteria (e.g. reader interest, romantic comedy vs
> > > > action
> > film)?
> > > >
> > > > Kerry
> > > >
> > > > -----Original Message-----
> > > > From: Wiki-research-l
> > > > [mailto:[hidden email]]
> > > > On Behalf Of Greg
> > > > Sent: Thursday, 22 August 2019 1:19 PM
> > > > To: [hidden email]
> > > > Subject: [Wiki-research-l] gender balance of wikipedia citations
> > > >
> > > > Greetings!
> > > >
> > > > I was looking for information about the gender balance of
> > > > Wikipedia citations and no one I've asked knows of any work on
> > > > this topic. Do
> > you?
> > > >
> > > > I think this is an important question.
> > > >
> > > > Here's what I've learned so far:
> > > >
> > > > Wikipedia citations are currently in the form of text strings.
> > > > There is also an initiative to place citations in an annotated
> > > > structured repository (wikicite). I do not know the current status
> > > > of wikicite or if/when this could be used for this inquiry--either
> > > > to examine all, or a sensible subset of the citations.
> > > >
> > > > My perspective is that understanding the gender balance is
> > > > necessary and urgent. The balance could be better, the same, or
> > > > worse than the citation balances we already know, and the scale of
> > > > the
> > effect is quite large.
> > > >
> > > > Is this a line of inquiry that the wikimedia/wikicite community is
> > > > interested in pursuing? If so, what is the best way to get started?
> > > > Does the WMF have the resources and interest to look into this
> > > > matter
> > inhouse?
> > > >
> > > > Thanks for your thoughts.
> > > >
> > > > Greg
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > Subject: Digest Footer
> > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > > >
> > > > ------------------------------
> > > >
> > > > End of Wiki-research-l Digest, Vol 168, Issue 11
> > > > ************************************************
> > > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 13
> > ************************************************
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 17
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

WereSpielChequers-2
Hi Greg,

One of the major step changes in the early growth of the English Wikipedia
was when a bot called RamBot created stub articles on US places. I think
they were cited to the census. Others have created articles on rivers in
countries and various other topics by similar programmatic means. Nowadays
such article creation is unlikely to get consensus on the English
Wikipedia, but there are some languages which are very open to such
creations and have them by the million.

I'm not sure if the fastest updating of existing articles is automated or
just semiautomated. But looking at the bot requests page, it certainly
looks like some people are running such maintenance bots "updating GDP by
country" is a current bot request.
https://en.wikipedia.org/wiki/Wikipedia:Bot_requests.

I'm not sure how "the ease of a source for purposes of converting into a
table and generating a separate article for each row" relates to gender.
But i suspect "number of times cited in wikipedia" deserves less kudos than
"number of times cited in academia".

WSC

On Sun, 25 Aug 2019 at 05:22, Greg <[hidden email]> wrote:

> Thanks again, Kerry. I am hoping that someone with access to more resources
> (knowledge, support, etc) than I have will look into this.
>
> A few more thoughts/questions:
>
> 1. The link to the citation dataset from the Medium article ("What are the
> ten most cited sources on Wikipedia? Let’s ask the data.") is broken.
> 2. As far as I can tell, every named author in the top ten most cited
> sources on Wikipedia is male. One piece is by a working group
> 3. This line from the Medium piece struck me: "Many of these publications
> have been cited by Wikipedians across large series of articles using
> powerful bots and automated tools."
>
> Are citations being added by bots? I'm not sure that I understand that line
> correctly.
>
> Greg
>
>
>
>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Greg-2
In reply to this post by Greg-2
Thanks, WSC. All very interesting.

I've been thinking about Wiklpedia citations less in terms of kudos and
more in terms of a feedback loop. The cited sources get a significant
amount of attention (1 click per 200 pageviews is the number I saw
recently). When I imagine total Wikipedia traffic, that's huge. How many
students are finding sources this way? How many academics? And how many of
these citations are finding their way back into academic publications via
this mechanism?

Assuming this is happening to some degree, the gender imbalance of the
citations is also reflected. If the Wikipedia imbalance is the same as the
one in academia, that's one thing; if it is better on Wikipedia than it is
in academia, that's reason to celebrate; if the balance is worse, that's
concerning. In fact, if the gender imbalance conforms to my fears instead
of my hopes, and is magnified by the massive website traffic, I imagine it
could even explain the growth in the citation disparity researchers note in
their study of political science texts. (I link to that study in a previous
post; it was mentioned in the Washington Post recently)

There is a very real possibility that Wikipedia is making the citation
gender gap worse. I think we need to understand what is happening and take
immediate action if the news is not good.

Greg

>
>
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Jane Darnell
Greg,
Thanks for worrying. This is a known problem and yes, Wikipedia contributes
to the Gendergap in citations and no, it's not an easy fix, since it is the
fault of systemic bias in academia. So fewer women are head author on
scientific publications, and it is generally only the head author that gets
cited on Wikipedia. This is not just a problem with written works in the
field of politics.  I spend most of my time working on paintings and their
documented catalogs, so generally I only notice and fix this problem in art
catalogs. Women rarely appear as lead author mentioned. I will always add
them in to descriptions when I add items for their works on Wikidata, but I
can not always find them! Sometimes I can't even create items for them
because all I have is a name and a work and nothing else available online
anywhere. You see this most often with women who spent entire careers
working at a single institution and the institution doesn't bother to
promote their work or even list them in exhibition catalogs. With luck
there might be a local obituary, but not always. If you have suggestions
how to set up a Wikiproject to tackle this it would be a good idea. In my
onwiki experience the Women-in-Red community can be very positive in their
response to gendergap-related issues for women writers.
Jane

On Mon, Aug 26, 2019 at 6:17 AM Greg <[hidden email]> wrote:

> Thanks, WSC. All very interesting.
>
> I've been thinking about Wiklpedia citations less in terms of kudos and
> more in terms of a feedback loop. The cited sources get a significant
> amount of attention (1 click per 200 pageviews is the number I saw
> recently). When I imagine total Wikipedia traffic, that's huge. How many
> students are finding sources this way? How many academics? And how many of
> these citations are finding their way back into academic publications via
> this mechanism?
>
> Assuming this is happening to some degree, the gender imbalance of the
> citations is also reflected. If the Wikipedia imbalance is the same as the
> one in academia, that's one thing; if it is better on Wikipedia than it is
> in academia, that's reason to celebrate; if the balance is worse, that's
> concerning. In fact, if the gender imbalance conforms to my fears instead
> of my hopes, and is magnified by the massive website traffic, I imagine it
> could even explain the growth in the citation disparity researchers note in
> their study of political science texts. (I link to that study in a previous
> post; it was mentioned in the Washington Post recently)
>
> There is a very real possibility that Wikipedia is making the citation
> gender gap worse. I think we need to understand what is happening and take
> immediate action if the news is not good.
>
> Greg
>
> >
> >
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Federico Leva (Nemo)
In reply to this post by Greg-2
Greg, 22/08/19 06:19:
> I do not know the current status of wikicite or if/when this
> could be used for this inquiry--either to examine all, or a sensible subset
> of the citations.

If I see correctly, you still did not receive an answer on the data
available.

It's true that the Figshare item for
<https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia>
was deleted (I've asked about it on the talk page), but it's trivial to
run https://pypi.org/project/mwcites/ and extract the data yourself, at
least for citations which use an identifier.

Some example datasets produced this way:
https://zenodo.org/record/15871
https://zenodo.org/record/55004
https://zenodo.org/record/54799

Once you extract the list of works, the fun begins. You'll need to
intersect with other data sources (Wikidata, ORCID, other?) and account
for a number of factors until you manage to find a subset of the data
which has a sufficiently high signal:noise ratio. For instance you might
need to filter or normalise by
* year of publication (some year recent enough to have good data but old
enough to allow the work to be cited elsewhere, be archived after embargos);
* country or institution (some probably have better ORCID coverage);
* field/discipline and language;
* open access status (per Unpaywall);
* number of expected pageviews and clicks (for instance using
<https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
<https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases>;
a link from 10k articles on asteroids or proteins is not the same as
being the lone link from a popular article which is not the same as a
link buried among a thousand others on a big article);
* time or duration of the addition (with one of the various diff
extraction libraries, content persistence data or possibly historical
eventstream if such a thing is available).

To avoid having to invent everything yourself, maybe you can reuse the
method of some similar study, for instance the one on the open access
citation advantage or one of the many which studied the gender imbalance
of citations and peer review in journals.

However, it's very possible that the noise is just too much for a
general computational method. You might consider a more manual approach
on a sample of relevant events, for instance the *removal* of citations,
which is in my opinion more significant than the addition.* You might
extract all the diffs which removed a citation from an article in the
last N years (probably they'll be in the order of 10^5 rather than
10^6), remove some massive events or outliers, sample 500-1000 of them
randomly and verify the required data manually.

As usual it will be impossible to have an objective assessment of
whether that citation was really (in)appropriate in that context
according to the (English or whatever) Wikipedia guidelines. To test
that too, you should replicate one of the various studies of the gender
imbalance of peer review, perhaps one of those which tried to assess the
impact of a double blind peer review system on the gender imbalance.
However, because the sources are already published, you'd need to
provide the agendered information yourself and make sure the
participants perform their assessment in some controlled environment
where they don't have access to any gendered information (i.e. where you
cut them off the internet).

How many years do you have to work on this project? :-)

Federico

(*) I might add a citation just because it's the first result a popular
search engine gives me, after glancing at the abstract and maybe the
journal home page; but if I remove an existing citation, hopefully I've
at least assessed its content and made a judgement about it, apart from
cases of mass removals for specific problems with certain articles or
publication venues.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Isaac Johnson
Regarding data, I have not been a part of these projects but I think that I
can help a bit with working links:
* The (I believe) original dataset can also be found here:
https://analytics.wikimedia.org/datasets/archive/public-datasets/all/mwrefs/
* A newer version of this dataset was produced that also included
information about whether the source was openly available and its topic:
** Meta page:
https://meta.wikimedia.org/wiki/Research:Towards_Modeling_Citation_Quality
** Figshare:
https://figshare.com/articles/Accessibility_and_topics_of_citations_with_identifiers_in_Wikipedia/6819710

On Mon, Aug 26, 2019 at 3:53 AM Federico Leva (Nemo) <[hidden email]>
wrote:

> Greg, 22/08/19 06:19:
> > I do not know the current status of wikicite or if/when this
> > could be used for this inquiry--either to examine all, or a sensible
> subset
> > of the citations.
>
> If I see correctly, you still did not receive an answer on the data
> available.
>
> It's true that the Figshare item for
> <
> https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia>
>
> was deleted (I've asked about it on the talk page), but it's trivial to
> run https://pypi.org/project/mwcites/ and extract the data yourself, at
> least for citations which use an identifier.
>
> Some example datasets produced this way:
> https://zenodo.org/record/15871
> https://zenodo.org/record/55004
> https://zenodo.org/record/54799
>
> Once you extract the list of works, the fun begins. You'll need to
> intersect with other data sources (Wikidata, ORCID, other?) and account
> for a number of factors until you manage to find a subset of the data
> which has a sufficiently high signal:noise ratio. For instance you might
> need to filter or normalise by
> * year of publication (some year recent enough to have good data but old
> enough to allow the work to be cited elsewhere, be archived after
> embargos);
> * country or institution (some probably have better ORCID coverage);
> * field/discipline and language;
> * open access status (per Unpaywall);
> * number of expected pageviews and clicks (for instance using
> <https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
> <https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases>;
>
> a link from 10k articles on asteroids or proteins is not the same as
> being the lone link from a popular article which is not the same as a
> link buried among a thousand others on a big article);
> * time or duration of the addition (with one of the various diff
> extraction libraries, content persistence data or possibly historical
> eventstream if such a thing is available).
>
> To avoid having to invent everything yourself, maybe you can reuse the
> method of some similar study, for instance the one on the open access
> citation advantage or one of the many which studied the gender imbalance
> of citations and peer review in journals.
>
> However, it's very possible that the noise is just too much for a
> general computational method. You might consider a more manual approach
> on a sample of relevant events, for instance the *removal* of citations,
> which is in my opinion more significant than the addition.* You might
> extract all the diffs which removed a citation from an article in the
> last N years (probably they'll be in the order of 10^5 rather than
> 10^6), remove some massive events or outliers, sample 500-1000 of them
> randomly and verify the required data manually.
>
> As usual it will be impossible to have an objective assessment of
> whether that citation was really (in)appropriate in that context
> according to the (English or whatever) Wikipedia guidelines. To test
> that too, you should replicate one of the various studies of the gender
> imbalance of peer review, perhaps one of those which tried to assess the
> impact of a double blind peer review system on the gender imbalance.
> However, because the sources are already published, you'd need to
> provide the agendered information yourself and make sure the
> participants perform their assessment in some controlled environment
> where they don't have access to any gendered information (i.e. where you
> cut them off the internet).
>
> How many years do you have to work on this project? :-)
>
> Federico
>
> (*) I might add a citation just because it's the first result a popular
> search engine gives me, after glancing at the abstract and maybe the
> journal home page; but if I remove an existing citation, hopefully I've
> at least assessed its content and made a judgement about it, apart from
> cases of mass removals for specific problems with certain articles or
> publication venues.
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>


--
Isaac Johnson -- Research Scientist -- Wikimedia Foundation
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Greg-2
In reply to this post by Greg-2
Thanks for sharing your experience and thoughts, Jane. I did not know this
was happening--I'm hardly an expert, so that's not surprising, and yet it's
still very troubling to hear. I'm not sure what you mean by setting up a
Wikiproject. Do you mean of ways for how to study this gap--i.e., the ideas
that have been floated in this thread to this point? Or are you thinking of
something else?

Greg

On Mon, Aug 26, 2019 at 5:00 AM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. Re: gender balance of Wikipedia citations (WereSpielChequers)
>    2. Re: gender balance of Wikipedia citations (Greg)
>    3. Re: sockpuppets and how to find them sooner (Federico Leva (Nemo))
>    4. Re: gender balance of Wikipedia citations (Jane Darnell)
>    5. Re: gender balance of wikipedia citations (Federico Leva (Nemo))
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 25 Aug 2019 14:28:25 +0100
> From: WereSpielChequers <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> Message-ID:
>         <CAAanWP3qJnMpLB4tr9Eqt4EJLg2kCihkb50UY-d8=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Hi Greg,
>
> One of the major step changes in the early growth of the English Wikipedia
> was when a bot called RamBot created stub articles on US places. I think
> they were cited to the census. Others have created articles on rivers in
> countries and various other topics by similar programmatic means. Nowadays
> such article creation is unlikely to get consensus on the English
> Wikipedia, but there are some languages which are very open to such
> creations and have them by the million.
>
> I'm not sure if the fastest updating of existing articles is automated or
> just semiautomated. But looking at the bot requests page, it certainly
> looks like some people are running such maintenance bots "updating GDP by
> country" is a current bot request.
> https://en.wikipedia.org/wiki/Wikipedia:Bot_requests.
>
> I'm not sure how "the ease of a source for purposes of converting into a
> table and generating a separate article for each row" relates to gender.
> But i suspect "number of times cited in wikipedia" deserves less kudos than
> "number of times cited in academia".
>
> WSC
>
> On Sun, 25 Aug 2019 at 05:22, Greg <[hidden email]> wrote:
>
> > Thanks again, Kerry. I am hoping that someone with access to more
> resources
> > (knowledge, support, etc) than I have will look into this.
> >
> > A few more thoughts/questions:
> >
> > 1. The link to the citation dataset from the Medium article ("What are
> the
> > ten most cited sources on Wikipedia? Let’s ask the data.") is broken.
> > 2. As far as I can tell, every named author in the top ten most cited
> > sources on Wikipedia is male. One piece is by a working group
> > 3. This line from the Medium piece struck me: "Many of these publications
> > have been cited by Wikipedians across large series of articles using
> > powerful bots and automated tools."
> >
> > Are citations being added by bots? I'm not sure that I understand that
> line
> > correctly.
> >
> > Greg
> >
> >
> >
> >
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 25 Aug 2019 21:16:25 -0700
> From: Greg <[hidden email]>
> To: [hidden email]
> Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> Message-ID:
>         <CAOO9DNvGyfvJkzyRq60cSQi-T80mAkUa=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Thanks, WSC. All very interesting.
>
> I've been thinking about Wiklpedia citations less in terms of kudos and
> more in terms of a feedback loop. The cited sources get a significant
> amount of attention (1 click per 200 pageviews is the number I saw
> recently). When I imagine total Wikipedia traffic, that's huge. How many
> students are finding sources this way? How many academics? And how many of
> these citations are finding their way back into academic publications via
> this mechanism?
>
> Assuming this is happening to some degree, the gender imbalance of the
> citations is also reflected. If the Wikipedia imbalance is the same as the
> one in academia, that's one thing; if it is better on Wikipedia than it is
> in academia, that's reason to celebrate; if the balance is worse, that's
> concerning. In fact, if the gender imbalance conforms to my fears instead
> of my hopes, and is magnified by the massive website traffic, I imagine it
> could even explain the growth in the citation disparity researchers note in
> their study of political science texts. (I link to that study in a previous
> post; it was mentioned in the Washington Post recently)
>
> There is a very real possibility that Wikipedia is making the citation
> gender gap worse. I think we need to understand what is happening and take
> immediate action if the news is not good.
>
> Greg
>
> >
> >
> >
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 26 Aug 2019 10:59:07 +0300
> From: "Federico Leva (Nemo)" <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>, Aaron Halfaker
>         <[hidden email]>, Kerry Raymond <[hidden email]>
> Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> Please everyone avoid using jargon specific to the English Wikipedia on
> this cross-language and cross-wiki mailing list.
>
> Aaron Halfaker, 23/08/19 17:36:
> > I think embeddings[1] would be a nice way to create a signature.
>
> There is some discussion of acceptable user fingerprinting (presumably
> to be available to CheckUsers only), other than the usual over-reliance
> on IP addresses, in particular at
> <
> https://meta.wikimedia.org/wiki/Talk:IP_Editing:_Privacy_Enhancement_and_Abuse_Mitigation
> >.
>
> Federico
>
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 26 Aug 2019 10:17:46 +0200
> From: Jane Darnell <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> Message-ID:
>         <CAFVcA-G87k26nBMr=-e-+C8o6eG0KQvVihH=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Greg,
> Thanks for worrying. This is a known problem and yes, Wikipedia contributes
> to the Gendergap in citations and no, it's not an easy fix, since it is the
> fault of systemic bias in academia. So fewer women are head author on
> scientific publications, and it is generally only the head author that gets
> cited on Wikipedia. This is not just a problem with written works in the
> field of politics.  I spend most of my time working on paintings and their
> documented catalogs, so generally I only notice and fix this problem in art
> catalogs. Women rarely appear as lead author mentioned. I will always add
> them in to descriptions when I add items for their works on Wikidata, but I
> can not always find them! Sometimes I can't even create items for them
> because all I have is a name and a work and nothing else available online
> anywhere. You see this most often with women who spent entire careers
> working at a single institution and the institution doesn't bother to
> promote their work or even list them in exhibition catalogs. With luck
> there might be a local obituary, but not always. If you have suggestions
> how to set up a Wikiproject to tackle this it would be a good idea. In my
> onwiki experience the Women-in-Red community can be very positive in their
> response to gendergap-related issues for women writers.
> Jane
>
> On Mon, Aug 26, 2019 at 6:17 AM Greg <[hidden email]> wrote:
>
> > Thanks, WSC. All very interesting.
> >
> > I've been thinking about Wiklpedia citations less in terms of kudos and
> > more in terms of a feedback loop. The cited sources get a significant
> > amount of attention (1 click per 200 pageviews is the number I saw
> > recently). When I imagine total Wikipedia traffic, that's huge. How many
> > students are finding sources this way? How many academics? And how many
> of
> > these citations are finding their way back into academic publications via
> > this mechanism?
> >
> > Assuming this is happening to some degree, the gender imbalance of the
> > citations is also reflected. If the Wikipedia imbalance is the same as
> the
> > one in academia, that's one thing; if it is better on Wikipedia than it
> is
> > in academia, that's reason to celebrate; if the balance is worse, that's
> > concerning. In fact, if the gender imbalance conforms to my fears instead
> > of my hopes, and is magnified by the massive website traffic, I imagine
> it
> > could even explain the growth in the citation disparity researchers note
> in
> > their study of political science texts. (I link to that study in a
> previous
> > post; it was mentioned in the Washington Post recently)
> >
> > There is a very real possibility that Wikipedia is making the citation
> > gender gap worse. I think we need to understand what is happening and
> take
> > immediate action if the news is not good.
> >
> > Greg
> >
> > >
> > >
> > >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 26 Aug 2019 11:45:09 +0300
> From: "Federico Leva (Nemo)" <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>, Greg
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> Greg, 22/08/19 06:19:
> > I do not know the current status of wikicite or if/when this
> > could be used for this inquiry--either to examine all, or a sensible
> subset
> > of the citations.
>
> If I see correctly, you still did not receive an answer on the data
> available.
>
> It's true that the Figshare item for
> <
> https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia>
>
> was deleted (I've asked about it on the talk page), but it's trivial to
> run https://pypi.org/project/mwcites/ and extract the data yourself, at
> least for citations which use an identifier.
>
> Some example datasets produced this way:
> https://zenodo.org/record/15871
> https://zenodo.org/record/55004
> https://zenodo.org/record/54799
>
> Once you extract the list of works, the fun begins. You'll need to
> intersect with other data sources (Wikidata, ORCID, other?) and account
> for a number of factors until you manage to find a subset of the data
> which has a sufficiently high signal:noise ratio. For instance you might
> need to filter or normalise by
> * year of publication (some year recent enough to have good data but old
> enough to allow the work to be cited elsewhere, be archived after
> embargos);
> * country or institution (some probably have better ORCID coverage);
> * field/discipline and language;
> * open access status (per Unpaywall);
> * number of expected pageviews and clicks (for instance using
> <https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
> <https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases>;
>
> a link from 10k articles on asteroids or proteins is not the same as
> being the lone link from a popular article which is not the same as a
> link buried among a thousand others on a big article);
> * time or duration of the addition (with one of the various diff
> extraction libraries, content persistence data or possibly historical
> eventstream if such a thing is available).
>
> To avoid having to invent everything yourself, maybe you can reuse the
> method of some similar study, for instance the one on the open access
> citation advantage or one of the many which studied the gender imbalance
> of citations and peer review in journals.
>
> However, it's very possible that the noise is just too much for a
> general computational method. You might consider a more manual approach
> on a sample of relevant events, for instance the *removal* of citations,
> which is in my opinion more significant than the addition.* You might
> extract all the diffs which removed a citation from an article in the
> last N years (probably they'll be in the order of 10^5 rather than
> 10^6), remove some massive events or outliers, sample 500-1000 of them
> randomly and verify the required data manually.
>
> As usual it will be impossible to have an objective assessment of
> whether that citation was really (in)appropriate in that context
> according to the (English or whatever) Wikipedia guidelines. To test
> that too, you should replicate one of the various studies of the gender
> imbalance of peer review, perhaps one of those which tried to assess the
> impact of a double blind peer review system on the gender imbalance.
> However, because the sources are already published, you'd need to
> provide the agendered information yourself and make sure the
> participants perform their assessment in some controlled environment
> where they don't have access to any gendered information (i.e. where you
> cut them off the internet).
>
> How many years do you have to work on this project? :-)
>
> Federico
>
> (*) I might add a citation just because it's the first result a popular
> search engine gives me, after glancing at the abstract and maybe the
> journal home page; but if I remove an existing citation, hopefully I've
> at least assessed its content and made a judgement about it, apart from
> cases of mass removals for specific problems with certain articles or
> publication venues.
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 20
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of wikipedia citations

Greg-2
In reply to this post by Isaac Johnson
Thanks, Isaac and Federico. These notes and links are very helpful--and
will require some time to process. As for how many years I have to work on
this, I'm retired! In truth, I keep hoping that someone on this list will
express interest in working on these matters. The questions are all very
interesting and quite relevant. The idea of studying removed citations is
both complex and compelling.

Greg

On Mon, Aug 26, 2019 at 6:49 AM Isaac Johnson <[hidden email]> wrote:

> Regarding data, I have not been a part of these projects but I think that
> I can help a bit with working links:
> * The (I believe) original dataset can also be found here:
> https://analytics.wikimedia.org/datasets/archive/public-datasets/all/mwrefs/
> * A newer version of this dataset was produced that also included
> information about whether the source was openly available and its topic:
> ** Meta page:
> https://meta.wikimedia.org/wiki/Research:Towards_Modeling_Citation_Quality
> ** Figshare:
> https://figshare.com/articles/Accessibility_and_topics_of_citations_with_identifiers_in_Wikipedia/6819710
>
> On Mon, Aug 26, 2019 at 3:53 AM Federico Leva (Nemo) <[hidden email]>
> wrote:
>
>> Greg, 22/08/19 06:19:
>> > I do not know the current status of wikicite or if/when this
>> > could be used for this inquiry--either to examine all, or a sensible
>> subset
>> > of the citations.
>>
>> If I see correctly, you still did not receive an answer on the data
>> available.
>>
>> It's true that the Figshare item for
>> <
>> https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia>
>>
>> was deleted (I've asked about it on the talk page), but it's trivial to
>> run https://pypi.org/project/mwcites/ and extract the data yourself, at
>> least for citations which use an identifier.
>>
>> Some example datasets produced this way:
>> https://zenodo.org/record/15871
>> https://zenodo.org/record/55004
>> https://zenodo.org/record/54799
>>
>> Once you extract the list of works, the fun begins. You'll need to
>> intersect with other data sources (Wikidata, ORCID, other?) and account
>> for a number of factors until you manage to find a subset of the data
>> which has a sufficiently high signal:noise ratio. For instance you might
>> need to filter or normalise by
>> * year of publication (some year recent enough to have good data but old
>> enough to allow the work to be cited elsewhere, be archived after
>> embargos);
>> * country or institution (some probably have better ORCID coverage);
>> * field/discipline and language;
>> * open access status (per Unpaywall);
>> * number of expected pageviews and clicks (for instance using
>> <https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
>> <https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases>;
>>
>> a link from 10k articles on asteroids or proteins is not the same as
>> being the lone link from a popular article which is not the same as a
>> link buried among a thousand others on a big article);
>> * time or duration of the addition (with one of the various diff
>> extraction libraries, content persistence data or possibly historical
>> eventstream if such a thing is available).
>>
>> To avoid having to invent everything yourself, maybe you can reuse the
>> method of some similar study, for instance the one on the open access
>> citation advantage or one of the many which studied the gender imbalance
>> of citations and peer review in journals.
>>
>> However, it's very possible that the noise is just too much for a
>> general computational method. You might consider a more manual approach
>> on a sample of relevant events, for instance the *removal* of citations,
>> which is in my opinion more significant than the addition.* You might
>> extract all the diffs which removed a citation from an article in the
>> last N years (probably they'll be in the order of 10^5 rather than
>> 10^6), remove some massive events or outliers, sample 500-1000 of them
>> randomly and verify the required data manually.
>>
>> As usual it will be impossible to have an objective assessment of
>> whether that citation was really (in)appropriate in that context
>> according to the (English or whatever) Wikipedia guidelines. To test
>> that too, you should replicate one of the various studies of the gender
>> imbalance of peer review, perhaps one of those which tried to assess the
>> impact of a double blind peer review system on the gender imbalance.
>> However, because the sources are already published, you'd need to
>> provide the agendered information yourself and make sure the
>> participants perform their assessment in some controlled environment
>> where they don't have access to any gendered information (i.e. where you
>> cut them off the internet).
>>
>> How many years do you have to work on this project? :-)
>>
>> Federico
>>
>> (*) I might add a citation just because it's the first result a popular
>> search engine gives me, after glancing at the abstract and maybe the
>> journal home page; but if I remove an existing citation, hopefully I've
>> at least assessed its content and made a judgement about it, apart from
>> cases of mass removals for specific problems with certain articles or
>> publication venues.
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
>
> --
> Isaac Johnson -- Research Scientist -- Wikimedia Foundation
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Jane Darnell
In reply to this post by Greg-2
Greg,
Yes that's what I meant. On Wikipedia you get what you measure, so many
Wikipedians are page-creators and page-hit junkies because we can measure
that. The trick to motivating editors is giving them other measurements for
progress. Here is the link to the Women writers Wikiproject and as you
scroll down you can see what is measured.
https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Women_writers
Jane

On Tue, Aug 27, 2019 at 3:39 AM Greg <[hidden email]> wrote:

> Thanks for sharing your experience and thoughts, Jane. I did not know this
> was happening--I'm hardly an expert, so that's not surprising, and yet it's
> still very troubling to hear. I'm not sure what you mean by setting up a
> Wikiproject. Do you mean of ways for how to study this gap--i.e., the ideas
> that have been floated in this thread to this point? Or are you thinking of
> something else?
>
> Greg
>
> On Mon, Aug 26, 2019 at 5:00 AM <
> [hidden email]>
> wrote:
>
> > Send Wiki-research-l mailing list submissions to
> >         [hidden email]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > or, via email, send a message with subject or body 'help' to
> >         [hidden email]
> >
> > You can reach the person managing the list at
> >         [hidden email]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Wiki-research-l digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: gender balance of Wikipedia citations (WereSpielChequers)
> >    2. Re: gender balance of Wikipedia citations (Greg)
> >    3. Re: sockpuppets and how to find them sooner (Federico Leva (Nemo))
> >    4. Re: gender balance of Wikipedia citations (Jane Darnell)
> >    5. Re: gender balance of wikipedia citations (Federico Leva (Nemo))
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Sun, 25 Aug 2019 14:28:25 +0100
> > From: WereSpielChequers <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> > Message-ID:
> >         <CAAanWP3qJnMpLB4tr9Eqt4EJLg2kCihkb50UY-d8=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Greg,
> >
> > One of the major step changes in the early growth of the English
> Wikipedia
> > was when a bot called RamBot created stub articles on US places. I think
> > they were cited to the census. Others have created articles on rivers in
> > countries and various other topics by similar programmatic means.
> Nowadays
> > such article creation is unlikely to get consensus on the English
> > Wikipedia, but there are some languages which are very open to such
> > creations and have them by the million.
> >
> > I'm not sure if the fastest updating of existing articles is automated or
> > just semiautomated. But looking at the bot requests page, it certainly
> > looks like some people are running such maintenance bots "updating GDP by
> > country" is a current bot request.
> > https://en.wikipedia.org/wiki/Wikipedia:Bot_requests.
> >
> > I'm not sure how "the ease of a source for purposes of converting into a
> > table and generating a separate article for each row" relates to gender.
> > But i suspect "number of times cited in wikipedia" deserves less kudos
> than
> > "number of times cited in academia".
> >
> > WSC
> >
> > On Sun, 25 Aug 2019 at 05:22, Greg <[hidden email]> wrote:
> >
> > > Thanks again, Kerry. I am hoping that someone with access to more
> > resources
> > > (knowledge, support, etc) than I have will look into this.
> > >
> > > A few more thoughts/questions:
> > >
> > > 1. The link to the citation dataset from the Medium article ("What are
> > the
> > > ten most cited sources on Wikipedia? Let’s ask the data.") is broken.
> > > 2. As far as I can tell, every named author in the top ten most cited
> > > sources on Wikipedia is male. One piece is by a working group
> > > 3. This line from the Medium piece struck me: "Many of these
> publications
> > > have been cited by Wikipedians across large series of articles using
> > > powerful bots and automated tools."
> > >
> > > Are citations being added by bots? I'm not sure that I understand that
> > line
> > > correctly.
> > >
> > > Greg
> > >
> > >
> > >
> > >
> > >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Sun, 25 Aug 2019 21:16:25 -0700
> > From: Greg <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> > Message-ID:
> >         <CAOO9DNvGyfvJkzyRq60cSQi-T80mAkUa=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Thanks, WSC. All very interesting.
> >
> > I've been thinking about Wiklpedia citations less in terms of kudos and
> > more in terms of a feedback loop. The cited sources get a significant
> > amount of attention (1 click per 200 pageviews is the number I saw
> > recently). When I imagine total Wikipedia traffic, that's huge. How many
> > students are finding sources this way? How many academics? And how many
> of
> > these citations are finding their way back into academic publications via
> > this mechanism?
> >
> > Assuming this is happening to some degree, the gender imbalance of the
> > citations is also reflected. If the Wikipedia imbalance is the same as
> the
> > one in academia, that's one thing; if it is better on Wikipedia than it
> is
> > in academia, that's reason to celebrate; if the balance is worse, that's
> > concerning. In fact, if the gender imbalance conforms to my fears instead
> > of my hopes, and is magnified by the massive website traffic, I imagine
> it
> > could even explain the growth in the citation disparity researchers note
> in
> > their study of political science texts. (I link to that study in a
> previous
> > post; it was mentioned in the Washington Post recently)
> >
> > There is a very real possibility that Wikipedia is making the citation
> > gender gap worse. I think we need to understand what is happening and
> take
> > immediate action if the news is not good.
> >
> > Greg
> >
> > >
> > >
> > >
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Mon, 26 Aug 2019 10:59:07 +0300
> > From: "Federico Leva (Nemo)" <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>, Aaron Halfaker
> >         <[hidden email]>, Kerry Raymond <
> [hidden email]>
> > Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> > Message-ID: <[hidden email]>
> > Content-Type: text/plain; charset=utf-8; format=flowed
> >
> > Please everyone avoid using jargon specific to the English Wikipedia on
> > this cross-language and cross-wiki mailing list.
> >
> > Aaron Halfaker, 23/08/19 17:36:
> > > I think embeddings[1] would be a nice way to create a signature.
> >
> > There is some discussion of acceptable user fingerprinting (presumably
> > to be available to CheckUsers only), other than the usual over-reliance
> > on IP addresses, in particular at
> > <
> >
> https://meta.wikimedia.org/wiki/Talk:IP_Editing:_Privacy_Enhancement_and_Abuse_Mitigation
> > >.
> >
> > Federico
> >
> >
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Mon, 26 Aug 2019 10:17:46 +0200
> > From: Jane Darnell <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> > Message-ID:
> >         <CAFVcA-G87k26nBMr=-e-+C8o6eG0KQvVihH=
> > [hidden email]>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Greg,
> > Thanks for worrying. This is a known problem and yes, Wikipedia
> contributes
> > to the Gendergap in citations and no, it's not an easy fix, since it is
> the
> > fault of systemic bias in academia. So fewer women are head author on
> > scientific publications, and it is generally only the head author that
> gets
> > cited on Wikipedia. This is not just a problem with written works in the
> > field of politics.  I spend most of my time working on paintings and
> their
> > documented catalogs, so generally I only notice and fix this problem in
> art
> > catalogs. Women rarely appear as lead author mentioned. I will always add
> > them in to descriptions when I add items for their works on Wikidata,
> but I
> > can not always find them! Sometimes I can't even create items for them
> > because all I have is a name and a work and nothing else available online
> > anywhere. You see this most often with women who spent entire careers
> > working at a single institution and the institution doesn't bother to
> > promote their work or even list them in exhibition catalogs. With luck
> > there might be a local obituary, but not always. If you have suggestions
> > how to set up a Wikiproject to tackle this it would be a good idea. In my
> > onwiki experience the Women-in-Red community can be very positive in
> their
> > response to gendergap-related issues for women writers.
> > Jane
> >
> > On Mon, Aug 26, 2019 at 6:17 AM Greg <[hidden email]> wrote:
> >
> > > Thanks, WSC. All very interesting.
> > >
> > > I've been thinking about Wiklpedia citations less in terms of kudos and
> > > more in terms of a feedback loop. The cited sources get a significant
> > > amount of attention (1 click per 200 pageviews is the number I saw
> > > recently). When I imagine total Wikipedia traffic, that's huge. How
> many
> > > students are finding sources this way? How many academics? And how many
> > of
> > > these citations are finding their way back into academic publications
> via
> > > this mechanism?
> > >
> > > Assuming this is happening to some degree, the gender imbalance of the
> > > citations is also reflected. If the Wikipedia imbalance is the same as
> > the
> > > one in academia, that's one thing; if it is better on Wikipedia than it
> > is
> > > in academia, that's reason to celebrate; if the balance is worse,
> that's
> > > concerning. In fact, if the gender imbalance conforms to my fears
> instead
> > > of my hopes, and is magnified by the massive website traffic, I imagine
> > it
> > > could even explain the growth in the citation disparity researchers
> note
> > in
> > > their study of political science texts. (I link to that study in a
> > previous
> > > post; it was mentioned in the Washington Post recently)
> > >
> > > There is a very real possibility that Wikipedia is making the citation
> > > gender gap worse. I think we need to understand what is happening and
> > take
> > > immediate action if the news is not good.
> > >
> > > Greg
> > >
> > > >
> > > >
> > > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> >
> >
> > ------------------------------
> >
> > Message: 5
> > Date: Mon, 26 Aug 2019 11:45:09 +0300
> > From: "Federico Leva (Nemo)" <[hidden email]>
> > To: Research into Wikimedia content and communities
> >         <[hidden email]>, Greg
> >         <[hidden email]>
> > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > Message-ID: <[hidden email]>
> > Content-Type: text/plain; charset=utf-8; format=flowed
> >
> > Greg, 22/08/19 06:19:
> > > I do not know the current status of wikicite or if/when this
> > > could be used for this inquiry--either to examine all, or a sensible
> > subset
> > > of the citations.
> >
> > If I see correctly, you still did not receive an answer on the data
> > available.
> >
> > It's true that the Figshare item for
> > <
> >
> https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia
> >
> >
> > was deleted (I've asked about it on the talk page), but it's trivial to
> > run https://pypi.org/project/mwcites/ and extract the data yourself, at
> > least for citations which use an identifier.
> >
> > Some example datasets produced this way:
> > https://zenodo.org/record/15871
> > https://zenodo.org/record/55004
> > https://zenodo.org/record/54799
> >
> > Once you extract the list of works, the fun begins. You'll need to
> > intersect with other data sources (Wikidata, ORCID, other?) and account
> > for a number of factors until you manage to find a subset of the data
> > which has a sufficiently high signal:noise ratio. For instance you might
> > need to filter or normalise by
> > * year of publication (some year recent enough to have good data but old
> > enough to allow the work to be cited elsewhere, be archived after
> > embargos);
> > * country or institution (some probably have better ORCID coverage);
> > * field/discipline and language;
> > * open access status (per Unpaywall);
> > * number of expected pageviews and clicks (for instance using
> > <https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
> > <https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases
> >;
> >
> > a link from 10k articles on asteroids or proteins is not the same as
> > being the lone link from a popular article which is not the same as a
> > link buried among a thousand others on a big article);
> > * time or duration of the addition (with one of the various diff
> > extraction libraries, content persistence data or possibly historical
> > eventstream if such a thing is available).
> >
> > To avoid having to invent everything yourself, maybe you can reuse the
> > method of some similar study, for instance the one on the open access
> > citation advantage or one of the many which studied the gender imbalance
> > of citations and peer review in journals.
> >
> > However, it's very possible that the noise is just too much for a
> > general computational method. You might consider a more manual approach
> > on a sample of relevant events, for instance the *removal* of citations,
> > which is in my opinion more significant than the addition.* You might
> > extract all the diffs which removed a citation from an article in the
> > last N years (probably they'll be in the order of 10^5 rather than
> > 10^6), remove some massive events or outliers, sample 500-1000 of them
> > randomly and verify the required data manually.
> >
> > As usual it will be impossible to have an objective assessment of
> > whether that citation was really (in)appropriate in that context
> > according to the (English or whatever) Wikipedia guidelines. To test
> > that too, you should replicate one of the various studies of the gender
> > imbalance of peer review, perhaps one of those which tried to assess the
> > impact of a double blind peer review system on the gender imbalance.
> > However, because the sources are already published, you'd need to
> > provide the agendered information yourself and make sure the
> > participants perform their assessment in some controlled environment
> > where they don't have access to any gendered information (i.e. where you
> > cut them off the internet).
> >
> > How many years do you have to work on this project? :-)
> >
> > Federico
> >
> > (*) I might add a citation just because it's the first result a popular
> > search engine gives me, after glancing at the abstract and maybe the
> > journal home page; but if I remove an existing citation, hopefully I've
> > at least assessed its content and made a judgement about it, apart from
> > cases of mass removals for specific problems with certain articles or
> > publication venues.
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
> >
> > ------------------------------
> >
> > End of Wiki-research-l Digest, Vol 168, Issue 20
> > ************************************************
> >
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: gender balance of Wikipedia citations

Greg-2
In reply to this post by Greg-2
Hi Jane,

Thanks for the link. It's clear that there is a lot of work being done, and
even more left to do.

I've been thinking about what you said about second authors and was
wondering if instead of fixing it (or in addition to fixing it), it would
make sense to put some sort of tag on the page itself (like the ones I see
questioning notability or requests for additional citations). Something
along the lines of authors missing from a particular citation and how to
fix that, or no work by women cited in this article (if this is the case).
It strikes me that by fixing it yourself, you are doing great work, but
that maybe it also makes sense to spread awareness about these issues to
the broader editing community so more people are thinking about it/doing
it. At any rate, I thought I'd float the idea. Such a tag/the response (if
any), could also be interesting to study, though perhaps something like
this already exists and I'm just not aware of it, or perhaps there is good
reason not to do it.

All best,
Greg

On Tue, Aug 27, 2019 at 5:00 AM <[hidden email]>
wrote:

> Send Wiki-research-l mailing list submissions to
>         [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> or, via email, send a message with subject or body 'help' to
>         [hidden email]
>
> You can reach the person managing the list at
>         [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wiki-research-l digest..."
>
>
> Today's Topics:
>
>    1. Re: gender balance of wikipedia citations (Greg)
>    2. Re: gender balance of Wikipedia citations (Jane Darnell)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 26 Aug 2019 18:56:12 -0700
> From: Greg <[hidden email]>
> To: Isaac Johnson <[hidden email]>
> Cc: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> Message-ID:
>         <
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Thanks, Isaac and Federico. These notes and links are very helpful--and
> will require some time to process. As for how many years I have to work on
> this, I'm retired! In truth, I keep hoping that someone on this list will
> express interest in working on these matters. The questions are all very
> interesting and quite relevant. The idea of studying removed citations is
> both complex and compelling.
>
> Greg
>
> On Mon, Aug 26, 2019 at 6:49 AM Isaac Johnson <[hidden email]> wrote:
>
> > Regarding data, I have not been a part of these projects but I think that
> > I can help a bit with working links:
> > * The (I believe) original dataset can also be found here:
> >
> https://analytics.wikimedia.org/datasets/archive/public-datasets/all/mwrefs/
> > * A newer version of this dataset was produced that also included
> > information about whether the source was openly available and its topic:
> > ** Meta page:
> >
> https://meta.wikimedia.org/wiki/Research:Towards_Modeling_Citation_Quality
> > ** Figshare:
> >
> https://figshare.com/articles/Accessibility_and_topics_of_citations_with_identifiers_in_Wikipedia/6819710
> >
> > On Mon, Aug 26, 2019 at 3:53 AM Federico Leva (Nemo) <[hidden email]
> >
> > wrote:
> >
> >> Greg, 22/08/19 06:19:
> >> > I do not know the current status of wikicite or if/when this
> >> > could be used for this inquiry--either to examine all, or a sensible
> >> subset
> >> > of the citations.
> >>
> >> If I see correctly, you still did not receive an answer on the data
> >> available.
> >>
> >> It's true that the Figshare item for
> >> <
> >>
> https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia
> >
> >>
> >> was deleted (I've asked about it on the talk page), but it's trivial to
> >> run https://pypi.org/project/mwcites/ and extract the data yourself, at
> >> least for citations which use an identifier.
> >>
> >> Some example datasets produced this way:
> >> https://zenodo.org/record/15871
> >> https://zenodo.org/record/55004
> >> https://zenodo.org/record/54799
> >>
> >> Once you extract the list of works, the fun begins. You'll need to
> >> intersect with other data sources (Wikidata, ORCID, other?) and account
> >> for a number of factors until you manage to find a subset of the data
> >> which has a sufficiently high signal:noise ratio. For instance you might
> >> need to filter or normalise by
> >> * year of publication (some year recent enough to have good data but old
> >> enough to allow the work to be cited elsewhere, be archived after
> >> embargos);
> >> * country or institution (some probably have better ORCID coverage);
> >> * field/discipline and language;
> >> * open access status (per Unpaywall);
> >> * number of expected pageviews and clicks (for instance using
> >> <https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
> >> <
> https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases>;
> >>
> >> a link from 10k articles on asteroids or proteins is not the same as
> >> being the lone link from a popular article which is not the same as a
> >> link buried among a thousand others on a big article);
> >> * time or duration of the addition (with one of the various diff
> >> extraction libraries, content persistence data or possibly historical
> >> eventstream if such a thing is available).
> >>
> >> To avoid having to invent everything yourself, maybe you can reuse the
> >> method of some similar study, for instance the one on the open access
> >> citation advantage or one of the many which studied the gender imbalance
> >> of citations and peer review in journals.
> >>
> >> However, it's very possible that the noise is just too much for a
> >> general computational method. You might consider a more manual approach
> >> on a sample of relevant events, for instance the *removal* of citations,
> >> which is in my opinion more significant than the addition.* You might
> >> extract all the diffs which removed a citation from an article in the
> >> last N years (probably they'll be in the order of 10^5 rather than
> >> 10^6), remove some massive events or outliers, sample 500-1000 of them
> >> randomly and verify the required data manually.
> >>
> >> As usual it will be impossible to have an objective assessment of
> >> whether that citation was really (in)appropriate in that context
> >> according to the (English or whatever) Wikipedia guidelines. To test
> >> that too, you should replicate one of the various studies of the gender
> >> imbalance of peer review, perhaps one of those which tried to assess the
> >> impact of a double blind peer review system on the gender imbalance.
> >> However, because the sources are already published, you'd need to
> >> provide the agendered information yourself and make sure the
> >> participants perform their assessment in some controlled environment
> >> where they don't have access to any gendered information (i.e. where you
> >> cut them off the internet).
> >>
> >> How many years do you have to work on this project? :-)
> >>
> >> Federico
> >>
> >> (*) I might add a citation just because it's the first result a popular
> >> search engine gives me, after glancing at the abstract and maybe the
> >> journal home page; but if I remove an existing citation, hopefully I've
> >> at least assessed its content and made a judgement about it, apart from
> >> cases of mass removals for specific problems with certain articles or
> >> publication venues.
> >>
> >> _______________________________________________
> >> Wiki-research-l mailing list
> >> [hidden email]
> >> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >>
> >
> >
> > --
> > Isaac Johnson -- Research Scientist -- Wikimedia Foundation
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 27 Aug 2019 08:00:45 +0200
> From: Jane Darnell <[hidden email]>
> To: Research into Wikimedia content and communities
>         <[hidden email]>
> Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> Message-ID:
>         <CAFVcA-HqVicR0k65J4iox0PD=
> [hidden email]>
> Content-Type: text/plain; charset="UTF-8"
>
> Greg,
> Yes that's what I meant. On Wikipedia you get what you measure, so many
> Wikipedians are page-creators and page-hit junkies because we can measure
> that. The trick to motivating editors is giving them other measurements for
> progress. Here is the link to the Women writers Wikiproject and as you
> scroll down you can see what is measured.
> https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Women_writers
> Jane
>
> On Tue, Aug 27, 2019 at 3:39 AM Greg <[hidden email]> wrote:
>
> > Thanks for sharing your experience and thoughts, Jane. I did not know
> this
> > was happening--I'm hardly an expert, so that's not surprising, and yet
> it's
> > still very troubling to hear. I'm not sure what you mean by setting up a
> > Wikiproject. Do you mean of ways for how to study this gap--i.e., the
> ideas
> > that have been floated in this thread to this point? Or are you thinking
> of
> > something else?
> >
> > Greg
> >
> > On Mon, Aug 26, 2019 at 5:00 AM <
> > [hidden email]>
> > wrote:
> >
> > > Send Wiki-research-l mailing list submissions to
> > >         [hidden email]
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > >         https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > or, via email, send a message with subject or body 'help' to
> > >         [hidden email]
> > >
> > > You can reach the person managing the list at
> > >         [hidden email]
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Wiki-research-l digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > >    1. Re: gender balance of Wikipedia citations (WereSpielChequers)
> > >    2. Re: gender balance of Wikipedia citations (Greg)
> > >    3. Re: sockpuppets and how to find them sooner (Federico Leva
> (Nemo))
> > >    4. Re: gender balance of Wikipedia citations (Jane Darnell)
> > >    5. Re: gender balance of wikipedia citations (Federico Leva (Nemo))
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Sun, 25 Aug 2019 14:28:25 +0100
> > > From: WereSpielChequers <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> > > Message-ID:
> > >         <CAAanWP3qJnMpLB4tr9Eqt4EJLg2kCihkb50UY-d8=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Hi Greg,
> > >
> > > One of the major step changes in the early growth of the English
> > Wikipedia
> > > was when a bot called RamBot created stub articles on US places. I
> think
> > > they were cited to the census. Others have created articles on rivers
> in
> > > countries and various other topics by similar programmatic means.
> > Nowadays
> > > such article creation is unlikely to get consensus on the English
> > > Wikipedia, but there are some languages which are very open to such
> > > creations and have them by the million.
> > >
> > > I'm not sure if the fastest updating of existing articles is automated
> or
> > > just semiautomated. But looking at the bot requests page, it certainly
> > > looks like some people are running such maintenance bots "updating GDP
> by
> > > country" is a current bot request.
> > > https://en.wikipedia.org/wiki/Wikipedia:Bot_requests.
> > >
> > > I'm not sure how "the ease of a source for purposes of converting into
> a
> > > table and generating a separate article for each row" relates to
> gender.
> > > But i suspect "number of times cited in wikipedia" deserves less kudos
> > than
> > > "number of times cited in academia".
> > >
> > > WSC
> > >
> > > On Sun, 25 Aug 2019 at 05:22, Greg <[hidden email]> wrote:
> > >
> > > > Thanks again, Kerry. I am hoping that someone with access to more
> > > resources
> > > > (knowledge, support, etc) than I have will look into this.
> > > >
> > > > A few more thoughts/questions:
> > > >
> > > > 1. The link to the citation dataset from the Medium article ("What
> are
> > > the
> > > > ten most cited sources on Wikipedia? Let’s ask the data.") is broken.
> > > > 2. As far as I can tell, every named author in the top ten most cited
> > > > sources on Wikipedia is male. One piece is by a working group
> > > > 3. This line from the Medium piece struck me: "Many of these
> > publications
> > > > have been cited by Wikipedians across large series of articles using
> > > > powerful bots and automated tools."
> > > >
> > > > Are citations being added by bots? I'm not sure that I understand
> that
> > > line
> > > > correctly.
> > > >
> > > > Greg
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Sun, 25 Aug 2019 21:16:25 -0700
> > > From: Greg <[hidden email]>
> > > To: [hidden email]
> > > Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> > > Message-ID:
> > >         <CAOO9DNvGyfvJkzyRq60cSQi-T80mAkUa=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Thanks, WSC. All very interesting.
> > >
> > > I've been thinking about Wiklpedia citations less in terms of kudos and
> > > more in terms of a feedback loop. The cited sources get a significant
> > > amount of attention (1 click per 200 pageviews is the number I saw
> > > recently). When I imagine total Wikipedia traffic, that's huge. How
> many
> > > students are finding sources this way? How many academics? And how many
> > of
> > > these citations are finding their way back into academic publications
> via
> > > this mechanism?
> > >
> > > Assuming this is happening to some degree, the gender imbalance of the
> > > citations is also reflected. If the Wikipedia imbalance is the same as
> > the
> > > one in academia, that's one thing; if it is better on Wikipedia than it
> > is
> > > in academia, that's reason to celebrate; if the balance is worse,
> that's
> > > concerning. In fact, if the gender imbalance conforms to my fears
> instead
> > > of my hopes, and is magnified by the massive website traffic, I imagine
> > it
> > > could even explain the growth in the citation disparity researchers
> note
> > in
> > > their study of political science texts. (I link to that study in a
> > previous
> > > post; it was mentioned in the Washington Post recently)
> > >
> > > There is a very real possibility that Wikipedia is making the citation
> > > gender gap worse. I think we need to understand what is happening and
> > take
> > > immediate action if the news is not good.
> > >
> > > Greg
> > >
> > > >
> > > >
> > > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 3
> > > Date: Mon, 26 Aug 2019 10:59:07 +0300
> > > From: "Federico Leva (Nemo)" <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>, Aaron Halfaker
> > >         <[hidden email]>, Kerry Raymond <
> > [hidden email]>
> > > Subject: Re: [Wiki-research-l] sockpuppets and how to find them sooner
> > > Message-ID: <[hidden email]>
> > > Content-Type: text/plain; charset=utf-8; format=flowed
> > >
> > > Please everyone avoid using jargon specific to the English Wikipedia on
> > > this cross-language and cross-wiki mailing list.
> > >
> > > Aaron Halfaker, 23/08/19 17:36:
> > > > I think embeddings[1] would be a nice way to create a signature.
> > >
> > > There is some discussion of acceptable user fingerprinting (presumably
> > > to be available to CheckUsers only), other than the usual over-reliance
> > > on IP addresses, in particular at
> > > <
> > >
> >
> https://meta.wikimedia.org/wiki/Talk:IP_Editing:_Privacy_Enhancement_and_Abuse_Mitigation
> > > >.
> > >
> > > Federico
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 4
> > > Date: Mon, 26 Aug 2019 10:17:46 +0200
> > > From: Jane Darnell <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of Wikipedia citations
> > > Message-ID:
> > >         <CAFVcA-G87k26nBMr=-e-+C8o6eG0KQvVihH=
> > > [hidden email]>
> > > Content-Type: text/plain; charset="UTF-8"
> > >
> > > Greg,
> > > Thanks for worrying. This is a known problem and yes, Wikipedia
> > contributes
> > > to the Gendergap in citations and no, it's not an easy fix, since it is
> > the
> > > fault of systemic bias in academia. So fewer women are head author on
> > > scientific publications, and it is generally only the head author that
> > gets
> > > cited on Wikipedia. This is not just a problem with written works in
> the
> > > field of politics.  I spend most of my time working on paintings and
> > their
> > > documented catalogs, so generally I only notice and fix this problem in
> > art
> > > catalogs. Women rarely appear as lead author mentioned. I will always
> add
> > > them in to descriptions when I add items for their works on Wikidata,
> > but I
> > > can not always find them! Sometimes I can't even create items for them
> > > because all I have is a name and a work and nothing else available
> online
> > > anywhere. You see this most often with women who spent entire careers
> > > working at a single institution and the institution doesn't bother to
> > > promote their work or even list them in exhibition catalogs. With luck
> > > there might be a local obituary, but not always. If you have
> suggestions
> > > how to set up a Wikiproject to tackle this it would be a good idea. In
> my
> > > onwiki experience the Women-in-Red community can be very positive in
> > their
> > > response to gendergap-related issues for women writers.
> > > Jane
> > >
> > > On Mon, Aug 26, 2019 at 6:17 AM Greg <[hidden email]>
> wrote:
> > >
> > > > Thanks, WSC. All very interesting.
> > > >
> > > > I've been thinking about Wiklpedia citations less in terms of kudos
> and
> > > > more in terms of a feedback loop. The cited sources get a significant
> > > > amount of attention (1 click per 200 pageviews is the number I saw
> > > > recently). When I imagine total Wikipedia traffic, that's huge. How
> > many
> > > > students are finding sources this way? How many academics? And how
> many
> > > of
> > > > these citations are finding their way back into academic publications
> > via
> > > > this mechanism?
> > > >
> > > > Assuming this is happening to some degree, the gender imbalance of
> the
> > > > citations is also reflected. If the Wikipedia imbalance is the same
> as
> > > the
> > > > one in academia, that's one thing; if it is better on Wikipedia than
> it
> > > is
> > > > in academia, that's reason to celebrate; if the balance is worse,
> > that's
> > > > concerning. In fact, if the gender imbalance conforms to my fears
> > instead
> > > > of my hopes, and is magnified by the massive website traffic, I
> imagine
> > > it
> > > > could even explain the growth in the citation disparity researchers
> > note
> > > in
> > > > their study of political science texts. (I link to that study in a
> > > previous
> > > > post; it was mentioned in the Washington Post recently)
> > > >
> > > > There is a very real possibility that Wikipedia is making the
> citation
> > > > gender gap worse. I think we need to understand what is happening and
> > > take
> > > > immediate action if the news is not good.
> > > >
> > > > Greg
> > > >
> > > > >
> > > > >
> > > > >
> > > > _______________________________________________
> > > > Wiki-research-l mailing list
> > > > [hidden email]
> > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 5
> > > Date: Mon, 26 Aug 2019 11:45:09 +0300
> > > From: "Federico Leva (Nemo)" <[hidden email]>
> > > To: Research into Wikimedia content and communities
> > >         <[hidden email]>, Greg
> > >         <[hidden email]>
> > > Subject: Re: [Wiki-research-l] gender balance of wikipedia citations
> > > Message-ID: <[hidden email]>
> > > Content-Type: text/plain; charset=utf-8; format=flowed
> > >
> > > Greg, 22/08/19 06:19:
> > > > I do not know the current status of wikicite or if/when this
> > > > could be used for this inquiry--either to examine all, or a sensible
> > > subset
> > > > of the citations.
> > >
> > > If I see correctly, you still did not receive an answer on the data
> > > available.
> > >
> > > It's true that the Figshare item for
> > > <
> > >
> >
> https://meta.wikimedia.org/wiki/Research:Scholarly_article_citations_in_Wikipedia
> > >
> > >
> > > was deleted (I've asked about it on the talk page), but it's trivial to
> > > run https://pypi.org/project/mwcites/ and extract the data yourself,
> at
> > > least for citations which use an identifier.
> > >
> > > Some example datasets produced this way:
> > > https://zenodo.org/record/15871
> > > https://zenodo.org/record/55004
> > > https://zenodo.org/record/54799
> > >
> > > Once you extract the list of works, the fun begins. You'll need to
> > > intersect with other data sources (Wikidata, ORCID, other?) and account
> > > for a number of factors until you manage to find a subset of the data
> > > which has a sufficiently high signal:noise ratio. For instance you
> might
> > > need to filter or normalise by
> > > * year of publication (some year recent enough to have good data but
> old
> > > enough to allow the work to be cited elsewhere, be archived after
> > > embargos);
> > > * country or institution (some probably have better ORCID coverage);
> > > * field/discipline and language;
> > > * open access status (per Unpaywall);
> > > * number of expected pageviews and clicks (for instance using
> > > <https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews> and
> > > <
> https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream#Releases
> > >;
> > >
> > > a link from 10k articles on asteroids or proteins is not the same as
> > > being the lone link from a popular article which is not the same as a
> > > link buried among a thousand others on a big article);
> > > * time or duration of the addition (with one of the various diff
> > > extraction libraries, content persistence data or possibly historical
> > > eventstream if such a thing is available).
> > >
> > > To avoid having to invent everything yourself, maybe you can reuse the
> > > method of some similar study, for instance the one on the open access
> > > citation advantage or one of the many which studied the gender
> imbalance
> > > of citations and peer review in journals.
> > >
> > > However, it's very possible that the noise is just too much for a
> > > general computational method. You might consider a more manual approach
> > > on a sample of relevant events, for instance the *removal* of
> citations,
> > > which is in my opinion more significant than the addition.* You might
> > > extract all the diffs which removed a citation from an article in the
> > > last N years (probably they'll be in the order of 10^5 rather than
> > > 10^6), remove some massive events or outliers, sample 500-1000 of them
> > > randomly and verify the required data manually.
> > >
> > > As usual it will be impossible to have an objective assessment of
> > > whether that citation was really (in)appropriate in that context
> > > according to the (English or whatever) Wikipedia guidelines. To test
> > > that too, you should replicate one of the various studies of the gender
> > > imbalance of peer review, perhaps one of those which tried to assess
> the
> > > impact of a double blind peer review system on the gender imbalance.
> > > However, because the sources are already published, you'd need to
> > > provide the agendered information yourself and make sure the
> > > participants perform their assessment in some controlled environment
> > > where they don't have access to any gendered information (i.e. where
> you
> > > cut them off the internet).
> > >
> > > How many years do you have to work on this project? :-)
> > >
> > > Federico
> > >
> > > (*) I might add a citation just because it's the first result a popular
> > > search engine gives me, after glancing at the abstract and maybe the
> > > journal home page; but if I remove an existing citation, hopefully I've
> > > at least assessed its content and made a judgement about it, apart from
> > > cases of mass removals for specific problems with certain articles or
> > > publication venues.
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > Wiki-research-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> > >
> > >
> > > ------------------------------
> > >
> > > End of Wiki-research-l Digest, Vol 168, Issue 20
> > > ************************************************
> > >
> > _______________________________________________
> > Wiki-research-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
> >
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>
>
> ------------------------------
>
> End of Wiki-research-l Digest, Vol 168, Issue 22
> ************************************************
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
12