Re: Wiki Research Jounal?

classic Classic list List threaded Threaded
49 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Ed H. Chi
There has been a lot of talk about how to start a journal.  The real
issue in starting a journal is not the editorial board, or the way it
is published, or whether it will gather the citation impact.  The real
issue is READERSHIP.

If you can get people to read the journal, then it will have editors
wanting to serve the journal, and it will gather citation impact.

The reason why WikiSym is changing is for the same reason.  People are
not going to the conference!  I think the attendance has been below
100 for some time now.  That's not a sustainable number for the amount
of work that goes into organizing a conference.

---------------
Ed H. Chi, Staff Research Scientist, Google
CHI2012 Technical Program co-chair

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Joe Corneli-3
On Sun, Nov 4, 2012 at 6:14 PM, Ed H. Chi <[hidden email]> wrote:
> There has been a lot of talk about how to start a journal.  The real
> issue in starting a journal is not the editorial board, or the way it
> is published, or whether it will gather the citation impact.  The real
> issue is READERSHIP.

I like this observation.  A few natural follow up questions to people
here would be:

(1) Where do you currently read about wiki research?
(2) Where do you currently publish about wiki research?
(3) What's missing?

For me:

(1) I get a surprising amount of leads from conversations that happen
on this list, and I don't pay all that much attention to where I end
up grabbing the papers from in the end.

(2) I've published at WikiSym, and in (for instance) Proceedings of
the International Conference on Computational Science, or other
subject-specific conferences/workshops.  One of these papers was
picked up and republished by Digital Education Review.  I also
contributed to a paper that was published at Alt.CHI.

(3) For me, what seems most "missing" is a place to talk about the
future of research in a *productive* (not necessarily "scholarly")
way.  For some thoughts gleaned from this list, see
http://meta.wikimedia.org/wiki/Wiki_Research_Ideas#Rethinking_the_future_of_research
-- but where to continue things like this?  Not sure.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Juliana Bastos Marques
In reply to this post by Ed H. Chi
Ed and others, based on your observations, I'd like to pose a side question:

The impression that I get from many of these symposia (and journals) is that there is not much space for research concerning Wikipedia and Education, such as teaching methodologies, case studies and such, not on the side of hard-science chunks of data. I know of lots of other professors who are doing the same thing as myself, but I see not many places for exchanging our experiences (conference-wise, not online channels, which, franky, I don't think are working much). Do you feel there is good room to topics such as mine?

Thanks,
Juliana.


On Sun, Nov 4, 2012 at 4:14 PM, Ed H. Chi <[hidden email]> wrote:
There has been a lot of talk about how to start a journal.  The real
issue in starting a journal is not the editorial board, or the way it
is published, or whether it will gather the citation impact.  The real
issue is READERSHIP.

If you can get people to read the journal, then it will have editors
wanting to serve the journal, and it will gather citation impact.

The reason why WikiSym is changing is for the same reason.  People are
not going to the conference!  I think the attendance has been below
100 for some time now.  That's not a sustainable number for the amount
of work that goes into organizing a conference.

---------------
Ed H. Chi, Staff Research Scientist, Google
CHI2012 Technical Program co-chair

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l



--
www.domusaurea.org

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Pierre-Carl Langlais
In reply to this post by Joe Corneli-3

Yes, that's a good point. Nevertheless, readership is not an inherent  
quality. It depends on several factors, most of which are  
attractiveness (is the journal layout fine to read? does the selection  
brings up insightful content?), positioning (which scientific  
disciplines are concerned?) and mediatization (how do we come to  
guarantee a specific notability of the journal per se?).

To this extent, the journal has some valuable assets :
*Us :) There have been already a lot of discussion since september. So  
far, the issue is well-debated and well-explored.
*The Wikimedian communities, that comprise many academics.
*The Open Access communities that may well be interested in this kind  
of experiment. In the specific context of the Academic Spring, the  
journal may possibly receive some comment in the general press.
*Having the advantage of Eprint (wider access) without its drawback  
(economic model…).
*International scale. I have actually given some publicity to the  
concept on my French well-read blog : http://blogs.rue89.com/les-coulisses-de-wikipedia/2012/10/23/libre-acces-les-chercheurs-defendent-leurs-travaux-et-lesprit

So far, the editorial board may have a good latitude in managing (or  
not managing) to attract readership…

PCL

Le 4 nov. 12 à 20:00, Joe Corneli a écrit :

> On Sun, Nov 4, 2012 at 6:14 PM, Ed H. Chi <[hidden email]> wrote:
>> There has been a lot of talk about how to start a journal.  The real
>> issue in starting a journal is not the editorial board, or the way it
>> is published, or whether it will gather the citation impact.  The  
>> real
>> issue is READERSHIP.
>
> I like this observation.  A few natural follow up questions to people
> here would be:
>
> (1) Where do you currently read about wiki research?
> (2) Where do you currently publish about wiki research?
> (3) What's missing?
>
> For me:
>
> (1) I get a surprising amount of leads from conversations that happen
> on this list, and I don't pay all that much attention to where I end
> up grabbing the papers from in the end.
>
> (2) I've published at WikiSym, and in (for instance) Proceedings of
> the International Conference on Computational Science, or other
> subject-specific conferences/workshops.  One of these papers was
> picked up and republished by Digital Education Review.  I also
> contributed to a paper that was published at Alt.CHI.
>
> (3) For me, what seems most "missing" is a place to talk about the
> future of research in a *productive* (not necessarily "scholarly")
> way.  For some thoughts gleaned from this list, see
> http://meta.wikimedia.org/wiki/Wiki_Research_Ideas#Rethinking_the_future_of_research
> -- but where to continue things like this?  Not sure.
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Dariusz Jemielniak-3
In reply to this post by Ed H. Chi
actually, with our community, it is not. What other journals die for, we have sort of provided. This is why a Wiki journal may have a better chance than others, but only if it is prepared with the academic career paths and full proper code of conduct nuances considered (double-blind scholarly peer review, proper editorial board, PDFs with page numbers, etc.).

dj


On Sun, Nov 4, 2012 at 7:14 PM, Ed H. Chi <[hidden email]> wrote:
There has been a lot of talk about how to start a journal.  The real
issue in starting a journal is not the editorial board, or the way it
is published, or whether it will gather the citation impact.  The real
issue is READERSHIP.

If you can get people to read the journal, then it will have editors
wanting to serve the journal, and it will gather citation impact.

The reason why WikiSym is changing is for the same reason.  People are
not going to the conference!  I think the attendance has been below
100 for some time now.  That's not a sustainable number for the amount
of work that goes into organizing a conference.

---------------
Ed H. Chi, Staff Research Scientist, Google
CHI2012 Technical Program co-chair

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l





--

__________________________
dr hab. Dariusz Jemielniak
profesor zarządzania
kierownik katedry Zarządzania Międzynarodowego
i centrum badawczego CROW
Akademia Leona Koźmińskiego
http://www.crow.alk.edu.pl

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Ziko van Dijk-3
In reply to this post by Juliana Bastos Marques
This is exactly my problem nowadays. I am a historian and don't have
much to say about software and data mining, but would like to read
more about the humanities approach with regard to Wikipedia. Readers
experiences and expectations, Wikipedia in school and university etc.
Kind regards
Ziko


2012/11/4 Juliana Bastos Marques <[hidden email]>:
> Ed and others, based on your observations, I'd like to pose a side question:
>
> The impression that I get from many of these symposia (and journals) is that
> there is not much space for research concerning Wikipedia and Education,
> such as teaching methodologies, case studies and such, not on the side of
> hard-science chunks of data.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Jounal?

Kerry Raymond
In reply to this post by Pierre-Carl Langlais
So long as Google scholar is indexing the journal, I see no problem with
finding a readership.

How often do you start your quest for relevant literature by sitting down
with a set of journal titles? Yes, that's how we used to do it. But now
everyone just sits down in front of Google Scholar and types keywords and
relevant papers are found regardless of the journal they are published in.

Kerry

 


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Chitu Okoli
In reply to this post by Dariusz Jemielniak-3
I'll first say that I've never been on an editorial board, so my comments might be somewhat limited. Like my students, I learn best when I'm shown where I'm mistaken, so I would like to learn from you all!

On one hand, I agree that readership is very important. On the other hand, I don't believe it is the most critical issue. I think the most critical matter in starting a new journal is to *attract high-quality submissions* --that is, to get researchers who do high quality work to submit some of their best research to the journal. If that can be achieved, then a high readership is virtually guaranteed. Other benefits should readily follow: high citations, ISI listing, etc.

I've been reading all the comments on this thread carefully, and I see two distinct issues that are being mixed into one: (1) a new journal dedicated to research about wikis, and maybe also about related phenomena; and (2) a journal with new kind of wiki-based peer review system. I think it is best to rather keep these issues distinct.

First, why do we need a new journal dedicated to wiki research? I would think we don't want a new journal that publishes mainly low-quality research; we want a new journal that publishes high-quality research. So, is there such a need for wiki researchers? That is, do publishers of high-quality wiki-related research have a hard time finding high-quality journal outlets to publish their research? Based on the excellent wiki-based research published in a wide variety of journals, I don't think such a problem exists. So, why would a researcher with high-quality wiki research risk publishing their hard work in a new, unproven, even experimental journal? In my case, I have tenure, so I might consider taking such risks. However, many of my colleagues are working towards establishing their research careers, and I would definitely advise them against publishing their best work in anything but proven journals.

My point is not that a new journal cannot attract high-quality research; rather, my suspicion is that it can do so only if it is filling a void for high-quality research on topics that are difficult to publish in existing high-quality outlets. I'm yet to see this issue addressed in this discussion.

Second, concerning a new kind of wikified peer review: I think that such an experiment is very much worthwhile and should be attempted. However, from a scientific perspective, an experiment to test phenomenon W (wikified peer review) should control for all other possibly confounding phenomena to make sure that the end result is an accurate reflection of a proper test of phenomenon W. In this case, the risks of a new journal with a poorly justified research focus (as I argued above) is a major confound that blurs the results of testing for W. In short, I think the best way to test wikified peer review is to work with an existing journal that has already established its viability and ability to attract high-quality submissions.

The Journal of Peer Production has been mentioned as a target candidate, and their description of their peer review philosophy indicates that they might be quite open to such an experiment, if not with all papers, at least with some: http://peerproduction.net/peer-review/process/. However, it is still a new journal, and doesn't seem to have yet reached the state of releasing regular issues, so its newness might yet be a confound for testing W.

~ Chitu


Dariusz Jemielniak a écrit :
actually, with our community, it is not. What other journals die for, we have sort of provided. This is why a Wiki journal may have a better chance than others, but only if it is prepared with the academic career paths and full proper code of conduct nuances considered (double-blind scholarly peer review, proper editorial board, PDFs with page numbers, etc.).

dj


On Sun, Nov 4, 2012 at 7:14 PM, Ed H. Chi <[hidden email]> wrote:
There has been a lot of talk about how to start a journal.  The real
issue in starting a journal is not the editorial board, or the way it
is published, or whether it will gather the citation impact.  The real
issue is READERSHIP.

If you can get people to read the journal, then it will have editors
wanting to serve the journal, and it will gather citation impact.

The reason why WikiSym is changing is for the same reason.  People are
not going to the conference!  I think the attendance has been below
100 for some time now.  That's not a sustainable number for the amount
of work that goes into organizing a conference.


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal? - Double-blind vs. single-blind review

Chitu Okoli
In reply to this post by Dariusz Jemielniak-3
Although in theory double-blind review is superior to single-blind, in practice double-blind vs. single-blind review has very little to do with journal quality. It is much more a matter of disciplinary culture. (Single-blind: authors don't know who the reviewers are, but reviewers do know who authors are; Double-blind: neither authors nor reviewers know who each other are)

Yes, I am certainly aware of studies that show that double-blind reviewing does indeed reduce the bias towards publishing famous researchers and reduces the bias against publishing work by minority researchers and women. So, I believe that double-blind reviewing is indeed meaningful. However, my general observation is that the decision for a journal to be double-blind or single-blind is mainly a matter of disciplinary tradition. To make a very gross generalization, social science journals are generally double-blind, whereas natural science and mathematical science journals are generally single-blind. This observation is very relevant to this discussion, because the wiki-research-l community sits in between this divide. My perception is that 90% of people who post on this list are in information systems (double-blind), computer science (single-blind) or information science (either double- or single-blind).

If we can accept that single-blind journals can be considered as high-quality as well, then I feel a journal with wikified peer review would do much better being single-blind, especially if its subject matter is wiki-related topics. I understand that one of the primary reasons many journals decide against going double-blind is because of the ridiculous gymnastics that have to be undertaken in many cases to try to hide authors' identity. In computer science, where many researchers make their programs available on the Web, and much of their research concerns particular websites that they have developed, double-blinding would often be impossible if attempted--reviewers couldn't properly evaluate the work without knowing who created it. I think this is very much the case for a wiki-based peer review system for much wiki-related research.

Of course, the official reviewers should remain anonymous. (I know that open peer review has often been proposed--authors and reviewers know each others' identities--and has even been experimented with on several occasions, but it does not seem to be a quality improvement--I can post citations if anyone is interested.) It is much easier to hide the identity of the official reviewers than it is to hide that of the authors, and the benefits of single-blinding are substantial and proven.

~ Chitu


Dariusz Jemielniak a écrit :
actually, with our community, it is not. What other journals die for, we have sort of provided. This is why a Wiki journal may have a better chance than others, but only if it is prepared with the academic career paths and full proper code of conduct nuances considered (double-blind scholarly peer review, proper editorial board, PDFs with page numbers, etc.).

dj

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Ward Cunningham
In reply to this post by Chitu Okoli
I wonder if a better place to innovate might be in the conduct of research, rather than the reporting, review and publication of research?

While wiki speeds collaboration within a community, the research literature favors long-lasting contributions outside the community. Wiki or wiki-like collaboration tools might be better applied to stimulating and conducting high-quality research that will then usefully feed existing publications.

I assume here that Wikipedia and the like have been sufficiently successful to raise more research questions and to enable more research methods than this community has time and/or ability to pursue. (If topics were scarce then they might be usefully hoarded.)

Of course there are many ways to share methods and directions and lots have already been applied in this community. However, I get the impression that results still take months or years to produce. 

Aside: I have built a data mining tool and methodology, Exploratory Parsing, that can read all of Wikipedia in 10 seconds for a useful notion of "read".  I have also created a Federated Wiki that promotes wiki-like sharing without need for a common vision or agreed social norms. I intend to combine the two to make an experiment manager where setups are easily shared and where methodological improvements and/or research redirections easily build on other's work in progress.

What would our environment have to be like before our collaborative cycle time is reduced to days? 


On Nov 4, 2012, at 2:27 PM, Chitu Okoli wrote:

On one hand, I agree that readership is very important. On the other hand, I don't believe it is the most critical issue. I think the most critical matter in starting a new journal is to *attract high-quality submissions* --that is, to get researchers who do high quality work to submit some of their best research to the journal. If that can be achieved, then a high readership is virtually guaranteed. Other benefits should readily follow: high citations, ISI listing, etc.

I've been reading all the comments on this thread carefully, and I see two distinct issues that are being mixed into one: (1) a new journal dedicated to research about wikis, and maybe also about related phenomena; and (2) a journal with new kind of wiki-based peer review system. I think it is best to rather keep these issues distinct.


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Jack Park
Wondering further, I recently became acquainted with the Sage
Bioinformatics Synapse platform:

https://synapse.sagebase.org/

by way of a keynote at the O'Reilly Strata Rx conference by the Sage
president, Stephen Friend;

http://www.youtube.com/watch?v=m4Pvq4sldbQ

A later talk (apparently not online) by Erich Huang made it clear that
their goal is to move research to the open source, open access arena.
Erich spoke in terms of putting everything from research journals, to
data, to software used for analysis online (e.g. at GitHub), making it
available for continuous peer review, evaluation, forking and
evolution, to augment the way science is communicated.

That notion seems very much in the spirit of Ward's Wiki Way.

I am still investigating Synapse. I hope to know much more about it soon.

Jack

On Sun, Nov 4, 2012 at 4:57 PM, Ward Cunningham <[hidden email]> wrote:

> I wonder if a better place to innovate might be in the conduct of research,
> rather than the reporting, review and publication of research?
>
> While wiki speeds collaboration within a community, the research literature
> favors long-lasting contributions outside the community. Wiki or wiki-like
> collaboration tools might be better applied to stimulating and conducting
> high-quality research that will then usefully feed existing publications.
>
> I assume here that Wikipedia and the like have been sufficiently successful
> to raise more research questions and to enable more research methods than
> this community has time and/or ability to pursue. (If topics were scarce
> then they might be usefully hoarded.)
>
> Of course there are many ways to share methods and directions and lots have
> already been applied in this community. However, I get the impression that
> results still take months or years to produce.
>
> Aside: I have built a data mining tool and methodology, Exploratory Parsing,
> that can read all of Wikipedia in 10 seconds for a useful notion of "read".
> I have also created a Federated Wiki that promotes wiki-like sharing without
> need for a common vision or agreed social norms. I intend to combine the two
> to make an experiment manager where setups are easily shared and where
> methodological improvements and/or research redirections easily build on
> other's work in progress.
>
> What would our environment have to be like before our collaborative cycle
> time is reduced to days?
>
>
> On Nov 4, 2012, at 2:27 PM, Chitu Okoli wrote:
>
> On one hand, I agree that readership is very important. On the other hand, I
> don't believe it is the most critical issue. I think the most critical
> matter in starting a new journal is to *attract high-quality submissions*
> --that is, to get researchers who do high quality work to submit some of
> their best research to the journal. If that can be achieved, then a high
> readership is virtually guaranteed. Other benefits should readily follow:
> high citations, ISI listing, etc.
>
> I've been reading all the comments on this thread carefully, and I see two
> distinct issues that are being mixed into one: (1) a new journal dedicated
> to research about wikis, and maybe also about related phenomena; and (2) a
> journal with new kind of wiki-based peer review system. I think it is best
> to rather keep these issues distinct.
>
>
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Steven Walling-3
In reply to this post by Ward Cunningham
On Sun, Nov 4, 2012 at 4:57 PM, Ward Cunningham <[hidden email]> wrote:
Aside: I have built a data mining tool and methodology, Exploratory Parsing, that can read all of Wikipedia in 10 seconds for a useful notion of "read".  I have also created a Federated Wiki that promotes wiki-like sharing without need for a common vision or agreed social norms. I intend to combine the two to make an experiment manager where setups are easily shared and where methodological improvements and/or research redirections easily build on other's work in progress.

What would our environment have to be like before our collaborative cycle time is reduced to days? 

I would highly encourage researchers interested in collaborative systems to take a look at Federated Wiki. Collaboration among experts is a very interesting potential use case, considering the way the wiki handles data and visualization, as well as the git-like way is allows for collaboration. 

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Ward Cunningham
In reply to this post by Jack Park
I just watched Stephen Friend's presentation. 15 minutes well spent. Thanks. -- Ward


On Nov 4, 2012, at 5:13 PM, Jack Park wrote:

> Wondering further, I recently became acquainted with the Sage
> Bioinformatics Synapse platform:
>
> https://synapse.sagebase.org/
>
> by way of a keynote at the O'Reilly Strata Rx conference by the Sage
> president, Stephen Friend;
>
> http://www.youtube.com/watch?v=m4Pvq4sldbQ


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Pierre-Carl Langlais
In reply to this post by Ward Cunningham
Le 5 nov. 2012 à 01:57, Ward Cunningham <[hidden email]> a écrit :

I wonder if a better place to innovate might be in the conduct of research, rather than the reporting, review and publication of research?

While wiki speeds collaboration within a community, the research literature favors long-lasting contributions outside the community. Wiki or wiki-like collaboration tools might be better applied to stimulating and conducting high-quality research that will then usefully feed existing publications.

I may sound a bit overractive, but can't we do both? I would easily imagine the following two-way system:
*A wiki-laboratory, which hosts quick and less quick projects in progress. That could also include some reactive analysis by social scientists to recent wikimedian and collaborative knowledge phenomena.
*A wiki-journal, that should give a definitive and quotable form to the preceding researches and enlarge the discussions to wider disciplinary debates. By publishing external submissions it could give some new food for thought to the wiki-laboratory.
The relationships between the two structures would be a relevant analogy to the mutually-rewarding dialogue between the practical and theoretical side of scientific research.



On Nov 4, 2012, at 2:27 PM, Chitu Okoli wrote:

On one hand, I agree that readership is very important. On the other hand, I don't believe it is the most critical issue. I think the most critical matter in starting a new journal is to *attract high-quality submissions* --that is, to get researchers who do high quality work to submit some of their best research to the journal. If that can be achieved, then a high readership is virtually guaranteed. Other benefits should readily follow: high citations, ISI listing, etc.

I've been reading all the comments on this thread carefully, and I see two distinct issues that are being mixed into one: (1) a new journal dedicated to research about wikis, and maybe also about related phenomena; and (2) a journal with new kind of wiki-based peer review system. I think it is best to rather keep these issues distinct.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal? - Double-blind vs. single-blind review

Kerry Raymond-2
In reply to this post by Chitu Okoli
I think a compromise position is to use single-blind unless the authors request double-blind and are therefore prepared to undertake the "ridiculous gymnastics" required.

Certainly (in computer science) I would agree that it is very hard for any established researcher publishing in their normal field to successfully disguise their identify.

Sent from my iPad

On 05/11/2012, at 8:30 AM, "Chitu Okoli" <[hidden email]> wrote:

> Although in theory double-blind review is superior to single-blind, in practice double-blind vs. single-blind review has very little to do with journal quality. It is much more a matter of disciplinary culture. (Single-blind: authors don't know who the reviewers are, but reviewers do know who authors are; Double-blind: neither authors nor reviewers know who each other are)
>
> Yes, I am certainly aware of studies that show that double-blind reviewing does indeed reduce the bias towards publishing famous researchers and reduces the bias against publishing work by minority researchers and women. So, I believe that double-blind reviewing is indeed meaningful. However, my general observation is that the decision for a journal to be double-blind or single-blind is mainly a matter of disciplinary tradition. To make a very gross generalization, social science journals are generally double-blind, whereas natural science and mathematical science journals are generally single-blind. This observation is very relevant to this discussion, because the wiki-research-l community sits in between this divide. My perception is that 90% of people who post on this list are in information systems (double-blind), computer science (single-blind) or information science (either double- or single-blind).
>
> If we can accept that single-blind journals can be considered as high-quality as well, then I feel a journal with wikified peer review would do much better being single-blind, especially if its subject matter is wiki-related topics. I understand that one of the primary reasons many journals decide against going double-blind is because of the ridiculous gymnastics that have to be undertaken in many cases to try to hide authors' identity. In computer science, where many researchers make their programs available on the Web, and much of their research concerns particular websites that they have developed, double-blinding would often be impossible if attempted--reviewers couldn't properly evaluate the work without knowing who created it. I think this is very much the case for a wiki-based peer review system for much wiki-related research.
>
> Of course, the official reviewers should remain anonymous. (I know that open peer review has often been proposed--authors and reviewers know each others' identities--and has even been experimented with on several occasions, but it does not seem to be a quality improvement--I can post citations if anyone is interested.) It is much easier to hide the identity of the official reviewers than it is to hide that of the authors, and the benefits of single-blinding are substantial and proven.
>
> ~ Chitu
>
>
> Dariusz Jemielniak a écrit :
>> actually, with our community, it is not. What other journals die for, we have sort of provided. This is why a Wiki journal may have a better chance than others, but only if it is prepared with the academic career paths and full proper code of conduct nuances considered (double-blind scholarly peer review, proper editorial board, PDFs with page numbers, etc.).
>>
>> dj
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Jodi Schneider-3
In reply to this post by Ward Cunningham
On Mon, Nov 5, 2012 at 12:57 AM, Ward Cunningham <[hidden email]> wrote:
I wonder if a better place to innovate might be in the conduct of research, rather than the reporting, review and publication of research?

+1*

Regarding the existing conversation, if we want a journal, we need to ask what the purpose is.

I'd highly recommend Jason Priem's notion of the "decoupled journal" [1][2]. Jason points out that journals have been used for four main purposes, historically:
  1. Registration
  2. Archiving
  3. Dissemination
  4. Certification
Decoupling these functions is the way forward for scholarly communication. And it's already been happening -- with ArXiV, SSRN, Math Overflow, ... and new ways of measuring research impact [3]. So which function(s) matter most to us?

We can ask:

(1) What can wikis do for the registration, archiving, and/or dissemination functions, better than existing technologies?

(2) How can wikis contribute to altmetrics [3] used for certification functions?

(3) How can we as a community surface the most interesting and powerful research? What technologies do we need? What social habits do we need? 

(4) And, finally, how can our answers to #3 contribute to the certification we value? (Prestige, publications counting for tenure, ...)

I think rather than trying to create a high profile, high impact traditional journal, if we focused on these and similar questions, we would both move wiki research forward, and drive scientific communication itself forward.

-Jodi

* of course, reporting and doing research aren't an either/or -- they're closely related and one drives the other

[1] Jason Priem at Purdue:
slides
I've written a short summary here:

[2] Also a draft article called "Decoupling the scholarly journal" by Jason Priem and Bradley M. Hemminger, under review for the Frontiers in Computational Neuroscience special issue "Beyond open access: visions for open evaluation of scientific papers by post-publication peer review"



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

koltzenburg
+1 Jodi!

I agree it would be great to experiment on-site as you suggest

Claudia

On Mon, 5 Nov 2012 10:39:34 +0000, Jodi Schneider wrote

> On Mon, Nov 5, 2012 at 12:57 AM, Ward Cunningham <[hidden email]> wrote:
>
> > I wonder if a better place to innovate might be in the conduct of
> > research, rather than the reporting, review and publication of research?
> >
>
> +1*
>
> Regarding the existing conversation, if we want a journal, we need to ask
> what the purpose is.
>
> I'd highly recommend Jason Priem's notion of the "decoupled journal"
> [1][2]. Jason points out that journals have been used for four main
> purposes, historically:
>
>    1. Registration
>    2. Archiving
>    3. Dissemination
>    4. Certification
>
> Decoupling these functions is the way forward for scholarly communication.
> And it's already been happening -- with ArXiV, SSRN, Math Overflow, ... and
> new ways of measuring research impact [3]. So which function(s) matter most
> to us?
>
> We can ask:
>
> (1) What can wikis do for the registration, archiving, and/or dissemination
> functions, better than existing technologies?
>
> (2) How can wikis contribute to altmetrics [3] used for certification
> functions?
>
> (3) How can we as a community surface the most interesting and powerful
> research? What technologies do we need? What social habits do we need?
>
> (4) And, finally, how can our answers to #3 contribute to the certification
> we value? (Prestige, publications counting for tenure, ...)
>
> I think rather than trying to create a high profile, high impact
> traditional journal, if we focused on these and similar questions, we would
> both move wiki research forward, and drive scientific communication itself
> forward.
>
> -Jodi
>
> * of course, reporting and doing research aren't an either/or -- they're
> closely related and one drives the other
>
> [1] Jason Priem at Purdue:
> video
> http://youtu.be/OM22JuiWYgE
> slides
> https://docs.google.com/present/view?id=ddfg787c_362f465q2g5
> I've written a short summary here:
> http://jodischneider.com/blog/2012/11/04/altmetrics-can-help-surface-
> quality-content-jason-priem-on-the-decoupled-journal-as-the-achievable-
> future-of-scholarly-communication/
>
> [2] Also a draft article called "Decoupling the scholarly journal" by Jason
> Priem and Bradley M. Hemminger, under review for the Frontiers in
> Computational Neuroscience special issue "Beyond open access: visions for
> open evaluation of scientific papers by post-publication peer review"
> https://docs.google.com/document/d/1xDOy9GXXrUFc9TUIR2C470DTau8JEgZ9k-SMNIx5pb8/edit?
hl=en_US&authkey=CMeCqOYD
>
> [3] http://altmetrics.org


thanks & cheers,
Claudia
[hidden email]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Emilio J. Rodríguez-Posada
In reply to this post by Jodi Schneider-3
As a side consideration, I think that "science" is elitist, today. Obviously, there are some required rules to assure and assess what is sciencie and what is not, but we have the opportunity to open science to the world.

Until now people just consume science. We are in a historical moment to welcome and engage the entire world in science: proposing stuff to be researched, developing tools, extracting and curating data, and checking and peer-reviewing facts.

In the same way writing the world memory is a task for all us (Wikipedia), sciencie follows the same principles.

Open science, open research, open review, open data.

2012/11/5 Jodi Schneider <[hidden email]>
On Mon, Nov 5, 2012 at 12:57 AM, Ward Cunningham <[hidden email]> wrote:
I wonder if a better place to innovate might be in the conduct of research, rather than the reporting, review and publication of research?

+1*

Regarding the existing conversation, if we want a journal, we need to ask what the purpose is.

I'd highly recommend Jason Priem's notion of the "decoupled journal" [1][2]. Jason points out that journals have been used for four main purposes, historically:
  1. Registration
  2. Archiving
  3. Dissemination
  4. Certification
Decoupling these functions is the way forward for scholarly communication. And it's already been happening -- with ArXiV, SSRN, Math Overflow, ... and new ways of measuring research impact [3]. So which function(s) matter most to us?

We can ask:

(1) What can wikis do for the registration, archiving, and/or dissemination functions, better than existing technologies?

(2) How can wikis contribute to altmetrics [3] used for certification functions?

(3) How can we as a community surface the most interesting and powerful research? What technologies do we need? What social habits do we need? 

(4) And, finally, how can our answers to #3 contribute to the certification we value? (Prestige, publications counting for tenure, ...)

I think rather than trying to create a high profile, high impact traditional journal, if we focused on these and similar questions, we would both move wiki research forward, and drive scientific communication itself forward.

-Jodi

* of course, reporting and doing research aren't an either/or -- they're closely related and one drives the other

[1] Jason Priem at Purdue:
slides
I've written a short summary here:

[2] Also a draft article called "Decoupling the scholarly journal" by Jason Priem and Bradley M. Hemminger, under review for the Frontiers in Computational Neuroscience special issue "Beyond open access: visions for open evaluation of scientific papers by post-publication peer review"



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--
Emilio J. Rodríguez-Posada
http://LibreFind.org - The wiki search engine



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Joe Corneli-3
In reply to this post by Pierre-Carl Langlais
On Mon, Nov 5, 2012 at 1:59 AM, Pierre-Carl Langlais
<[hidden email]> wrote:

> I may sound a bit overractive, but can't we do both? I would easily imagine
> the following two-way system:
> *A wiki-laboratory, which hosts quick and less quick projects in progress.
> That could also include some reactive analysis by social scientists to
> recent wikimedian and collaborative knowledge phenomena.
> *A wiki-journal, that should give a definitive and quotable form to the
> preceding researches and enlarge the discussions to wider disciplinary
> debates. By publishing external submissions it could give some new food for
> thought to the wiki-laboratory.
> The relationships between the two structures would be a relevant analogy to
> the mutually-rewarding dialogue between the practical and theoretical side
> of scientific research.

I think we should be aware of two interesting "countervailing" trends.

(1) Observe that we are STILL having this conversation on a mailing
list, despite the existence of a wiki page that is in theory devoted
to exploring precisely these issues.
(http://meta.wikimedia.org/wiki/Wiki_Research_Ideas)

It would appear that working on a wiki is extra work compared to
working on a mailing list (at least for this "exploratory" phase of
the conversation).  That's not meant to be a normative judgment, it
just points to the need for a broader awareness of what wiki-like
research might look like.  In particular, it may also look like
"correspondence".  So, let's imagine that the wiki-laboratory was
already prototyped and tested on usenet.  What should our next steps
be?

(2) Observe that the wiki journal idea is connected with a particular
research obsession (for some), namely "finding experts".  Of course,
in order to find an expert, the expert must first have been created or
manufactured.  In a positive light, this means "education".  In a
negative light, it means regimes for producing stratification and
alienation.

Both, of course, exist already.  So again the question is one of next
steps, not something "de novo".  My sense is that that practicality is
usually abhorred (it's expensive, mundane, and you can't "get credit
for it"), whereas theory is strongly preferred (it's powerful,
efficient, and it can go on your CV).

In my view, the only sensible "solution" is to dissolve the (imagined)
separation between practice and theory, and look instead at the actual
practices of researchers and knowledge workers, trying to support them
(i.e. us), in what we actually do.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: Wiki Research Journal?

Pierre-Carl Langlais
In reply to this post by Emilio J. Rodríguez-Posada
Following the discussion of yesterday, I have enhanced a bit the design draft: http://meta.wikimedia.org/wiki/Wiki_Research_Ideas/Design

It now includes a specific thema for each issue. For instance, I have chosen « Wikipedia Verifiability ».

In order to visualize what kind of content we could get, I have replaced the "Lorem ipsum" stuff with « fake » summaries. 

As you can see, the journal may include both social science analyses (cf. the quellenkritik to wikipedia thing), and computer science experiment (the device from wikisource). 

Those who are interesting in taking part to the project may add their names here : http://meta.wikimedia.org/wiki/Wiki_Research_Ideas/Volunteers

Kind regards,

As a side consideration, I think that "science" is elitist, today. Obviously, there are some required rules to assure and assess what is sciencie and what is not, but we have the opportunity to open science to the world.

Until now people just consume science. We are in a historical moment to welcome and engage the entire world in science: proposing stuff to be researched, developing tools, extracting and curating data, and checking and peer-reviewing facts.

In the same way writing the world memory is a task for all us (Wikipedia), sciencie follows the same principles.

Open science, open research, open review, open data.

2012/11/5 Jodi Schneider <[hidden email]>
On Mon, Nov 5, 2012 at 12:57 AM, Ward Cunningham <[hidden email]> wrote:
I wonder if a better place to innovate might be in the conduct of research, rather than the reporting, review and publication of research?

+1*

Regarding the existing conversation, if we want a journal, we need to ask what the purpose is.

I'd highly recommend Jason Priem's notion of the "decoupled journal" [1][2]. Jason points out that journals have been used for four main purposes, historically:
  1. Registration
  2. Archiving
  3. Dissemination
  4. Certification
Decoupling these functions is the way forward for scholarly communication. And it's already been happening -- with ArXiV, SSRN, Math Overflow, ... and new ways of measuring research impact [3]. So which function(s) matter most to us?

We can ask:

(1) What can wikis do for the registration, archiving, and/or dissemination functions, better than existing technologies?

(2) How can wikis contribute to altmetrics [3] used for certification functions?

(3) How can we as a community surface the most interesting and powerful research? What technologies do we need? What social habits do we need? 

(4) And, finally, how can our answers to #3 contribute to the certification we value? (Prestige, publications counting for tenure, ...)

I think rather than trying to create a high profile, high impact traditional journal, if we focused on these and similar questions, we would both move wiki research forward, and drive scientific communication itself forward.

-Jodi

* of course, reporting and doing research aren't an either/or -- they're closely related and one drives the other

[1] Jason Priem at Purdue:
slides
I've written a short summary here:

[2] Also a draft article called "Decoupling the scholarly journal" by Jason Priem and Bradley M. Hemminger, under review for the Frontiers in Computational Neuroscience special issue "Beyond open access: visions for open evaluation of scientific papers by post-publication peer review"



_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l




--
Emilio J. Rodríguez-Posada
http://LibreFind.org - The wiki search engine


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
123