Exploring the Feasibility of Automatically Rating Online Article Quality

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Exploring the Feasibility of Automatically Rating Online Article Quality

Brian J Mingus
What would you do with a system that was intelligent enough to analyze a Wikipedia article along with qualitative human judgments of that article ("brilliant prose") and tell you exactly what the humans meant, even when they weren't sure themselves?

At least years Wikimania Erik Zachte and I discussed exactly this possibility. Invariably these discussions lead to the subjective nature of quality, and quickly diverge to [[Zen and the art of Motorcycle Maintenance]]. But my colleagues and I have determined that this problem should be tractable, and have initiated a research program to find out if we are correct.

What we would like to know is your dreams for such a system, what you would like it to do, and what you would do with it. To help kick-start your imaginations, please see the following paper, written by myself, a psychologist, Trevor Pincock, a linguist, and Laura Rassbach, a computer scientist. Although we consider our findings to be preliminary, we would also like to emphasize the phenomenal rate of growth of the field of [[Natural Language Processing]]. We *will* be able to build the system of your dreams. We just need to hear them first.

Please note that this paper is a draft. Please do not cite it.

"Exploring the Feasibility of Automatically Rating Online Article Quality"
http://whisper.colorado.edu/RassbachPincockMingus07.pdf

/Brian Mingus
en:User:Alterego



_______________________________________________
Wiki-research-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Exploring the Feasibility of Automatically RatingOnline Article Quality

alain_desilets
What's the killer scenario for such a system?
 
One possibility is a system that helps page "gardeners" in weeding out garbage contributions, or modifying them to be more appropriate. But I'm not sure that would be that useful. The track page feature seems good enough to allow page gardeners to follow changes being made and to quicky evaluate them.
 
Another possibility is something that would asses the page for "unity". For example, many pages contain entries that are written at different levels of formality. Rewriting those pages to make them more unified is hard work. So a tool that would help doing that might be useful.
 

----
Alain Désilets, National Research Council of Canada
Chair, WikiSym 2007

2007 International Symposium on Wikis
Wikis at Work in the World:
Open, Organic, Participatory Media for the 21st Century

http://www.wikisym.org/ws2007/

 
 
From: [hidden email] [mailto:[hidden email]] On Behalf Of Brian
Sent: May 9, 2007 12:38 AM
To: Research into Wikimedia content and communities
Subject: [Wiki-research-l] Exploring the Feasibility of Automatically RatingOnline Article Quality

What would you do with a system that was intelligent enough to analyze a Wikipedia article along with qualitative human judgments of that article ("brilliant prose") and tell you exactly what the humans meant, even when they weren't sure themselves?

At least years Wikimania Erik Zachte and I discussed exactly this possibility. Invariably these discussions lead to the subjective nature of quality, and quickly diverge to [[Zen and the art of Motorcycle Maintenance]]. But my colleagues and I have determined that this problem should be tractable, and have initiated a research program to find out if we are correct.

What we would like to know is your dreams for such a system, what you would like it to do, and what you would do with it. To help kick-start your imaginations, please see the following paper, written by myself, a psychologist, Trevor Pincock, a linguist, and Laura Rassbach, a computer scientist. Although we consider our findings to be preliminary, we would also like to emphasize the phenomenal rate of growth of the field of [[Natural Language Processing]]. We *will* be able to build the system of your dreams. We just need to hear them first.

Please note that this paper is a draft. Please do not cite it.

"Exploring the Feasibility of Automatically Rating Online Article Quality"
http://whisper.colorado.edu/RassbachPincockMingus07.pdf

/Brian Mingus
en:User:Alterego



_______________________________________________
Wiki-research-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Exploring the Feasibility of Automatically Rating Online Article Quality

Ward Cunningham
In reply to this post by Brian J Mingus
Interesting paper. 

I am building a model of wiki community behavior that talks in terms such as number of reads and writes under various circumstances. My model hypothesizes various mental states that are not so easily measured but might be possible to infer. I'm not far enough along to say much more about this now, except ...

In my dreams, I would find it useful to classify contributions into broad categories based on the nature of the writing such that I could get distinct counts for each. My expectation would be that this would simplify my inference of mental states, on average, even if individual classifications were not reliable.

Best regards. -- Ward



On May 8, 2007, at 9:38 PM, Brian wrote:

What we would like to know is your dreams for such a system, what you would like it to do, and what you would do with it. To help kick-start your imaginations, please see the following paper...

"Exploring the Feasibility of Automatically Rating Online Article Quality"
http://whisper.colorado.edu/RassbachPincockMingus07.pdf


_______________________________________________
Wiki-research-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Exploring the Feasibility of Automatically Rating Online Article Quality

Ee-Peng Lim
Ee-Peng wrote
I tried downloading the paper but the server seems to be disconnected.
Does anyone care to send me a copy?
Our group has also done some work in measuring article quality in Wikipedia.
It will be interesting to compare the differences in our approaches.

Ee-Peng Lim, Ba-Quy Vuong, Hady Wirawan Lauw and Aixin Sun. “Measuring Qualities of Articles Contributed by Online Communities”. In Proceedings of IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006), HongKong, Dec 2006.