[Wikimedia-l] Is Wikipedia Really 90% Wrong

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Wikimedia-l] Is Wikipedia Really 90% Wrong

jmh649
The journal article by Hasty et al published on May 1st 2014 basically took
ten Wikipedia articles and ten “researchers” (either medical students or
residents). Each Wikipedia article was then assessed by two of these
researchers to try to determine how many statements of fact they contained.
The first issue was that the number of statements of fact each reviewer
found sometimes differed by nearly 100%. They than took these individual
facts and the “researchers” compared them with the peer reviewed literature
as found on pubmed or the medical website Uptodate. They did not check to
see if the sources Wikipedia was using were high quality or were accurately
reflected. Additionally medical students and residents are hardly experts
in medical research.

No errors in Wikipedia are mentioned directly in the original journal
article. When I spoke with the lead author he declined to release the
underlying data for us at Wikipedia to correct the “errors” they had found
stating that he may 1) wish to publish more on the topic and 2) wished to
protect the researchers. So much for independent verifiability in science.
Hasty did make some claims to the popular press about errors on Wikipedia.
Some of the facts he mentioned however accurately reflected some of the
best available peer reviewed sources. For example he claimed that blood
pressure should only be checked twice to make the diagnosis of hypertension
and that when we state three times we are wrong. However look at the
National Institute for Health and Care Excellence (previous known as the
National Institute of Clinical Health / NICE) on page 7 in this document
http://www.nice.org.uk/nicemedia/pdf/CG18background.pdf It is thus a little
ironic that the Telegraph, a UK paper, repeated this incorrect statement
and the BBC covered the story so uncritically.

Wikipedia has strong recommendations for what counts as a suitable source.
We recommend the use of secondary sources published in well respected
journals from the last 3-5 years, position statements of national or
internationally recognized medical bodies or major textbooks. Is Wikipedia
a perfect source? No, but it is just as good as many and better than most
other sources out there. Or else why would the world be using it? Hasty's
work did not have a comparison group. Basically he invented a new method to
test the quality of medical content and then only applied this new method
to one source, Wikipedia. Without a comparator this single data point is
meaningless. I am curious what he would have found if he would have applied
this to a NICE guideline or emedicine?

We recently surveyed our top contributors and asked about their
backgrounds. What we found was that 52% have either a masters, PhD, or MD.
Another 33% have a BSc. About half are health care providers. 82% are male,
9% are female and 9% classified themselves as other or would rather not
say. This is very similar to results published by Nusa Faric in her
master's thesis. Additionally we are working with a number of organizations
including: the National Institute of Health, the Cochrane collaboration,
and the UCSF college of medicine among others to improve Wikipedia’s health
care content.

What Hasty did show was 1) the peer reviewed literature does not agree with
itself (ie different peer reviewed sources come to different conclusions
which is no surprise to anyone that has read much of it) 2) the peer review
process is sometimes flawed as he was able to publish a "peer reviewed"
article whose data does not support its conclusions. As someone who has
read a lot of the peer reviewed literature this is also not surprising.


--
James Heilman
MD, CCFP-EM, Wikipedian

The Wikipedia Open Textbook of Medicine
www.opentextbookofmedicine.com
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Is Wikipedia Really 90% Wrong

Jasper Deng
I am pretty sure that a "90% wrong" figure would fail an elementary
statistical test of significance...


On Sun, Jun 1, 2014 at 12:55 PM, James Heilman <[hidden email]> wrote:

> The journal article by Hasty et al published on May 1st 2014 basically took
> ten Wikipedia articles and ten “researchers” (either medical students or
> residents). Each Wikipedia article was then assessed by two of these
> researchers to try to determine how many statements of fact they contained.
> The first issue was that the number of statements of fact each reviewer
> found sometimes differed by nearly 100%. They than took these individual
> facts and the “researchers” compared them with the peer reviewed literature
> as found on pubmed or the medical website Uptodate. They did not check to
> see if the sources Wikipedia was using were high quality or were accurately
> reflected. Additionally medical students and residents are hardly experts
> in medical research.
>
> No errors in Wikipedia are mentioned directly in the original journal
> article. When I spoke with the lead author he declined to release the
> underlying data for us at Wikipedia to correct the “errors” they had found
> stating that he may 1) wish to publish more on the topic and 2) wished to
> protect the researchers. So much for independent verifiability in science.
> Hasty did make some claims to the popular press about errors on Wikipedia.
> Some of the facts he mentioned however accurately reflected some of the
> best available peer reviewed sources. For example he claimed that blood
> pressure should only be checked twice to make the diagnosis of hypertension
> and that when we state three times we are wrong. However look at the
> National Institute for Health and Care Excellence (previous known as the
> National Institute of Clinical Health / NICE) on page 7 in this document
> http://www.nice.org.uk/nicemedia/pdf/CG18background.pdf It is thus a
> little
> ironic that the Telegraph, a UK paper, repeated this incorrect statement
> and the BBC covered the story so uncritically.
>
> Wikipedia has strong recommendations for what counts as a suitable source.
> We recommend the use of secondary sources published in well respected
> journals from the last 3-5 years, position statements of national or
> internationally recognized medical bodies or major textbooks. Is Wikipedia
> a perfect source? No, but it is just as good as many and better than most
> other sources out there. Or else why would the world be using it? Hasty's
> work did not have a comparison group. Basically he invented a new method to
> test the quality of medical content and then only applied this new method
> to one source, Wikipedia. Without a comparator this single data point is
> meaningless. I am curious what he would have found if he would have applied
> this to a NICE guideline or emedicine?
>
> We recently surveyed our top contributors and asked about their
> backgrounds. What we found was that 52% have either a masters, PhD, or MD.
> Another 33% have a BSc. About half are health care providers. 82% are male,
> 9% are female and 9% classified themselves as other or would rather not
> say. This is very similar to results published by Nusa Faric in her
> master's thesis. Additionally we are working with a number of organizations
> including: the National Institute of Health, the Cochrane collaboration,
> and the UCSF college of medicine among others to improve Wikipedia’s health
> care content.
>
> What Hasty did show was 1) the peer reviewed literature does not agree with
> itself (ie different peer reviewed sources come to different conclusions
> which is no surprise to anyone that has read much of it) 2) the peer review
> process is sometimes flawed as he was able to publish a "peer reviewed"
> article whose data does not support its conclusions. As someone who has
> read a lot of the peer reviewed literature this is also not surprising.
>
>
> --
> James Heilman
> MD, CCFP-EM, Wikipedian
>
> The Wikipedia Open Textbook of Medicine
> www.opentextbookofmedicine.com
> _______________________________________________
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:[hidden email]?subject=unsubscribe>
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>