Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Michele Lavazza
Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much,
best regards,

Michele Lavazza
(Wikimania 2016 Programme Committee Deputy Chair)

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Lodewijk
Hi Michele,

lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option. 

And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all. 

Lodewijk

On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza <[hidden email]> wrote:
Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much,
best regards,

Michele Lavazza
(Wikimania 2016 Programme Committee Deputy Chair)

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Michele Lavazza
Sure, we will upload the comments and feedback of pc members to the Wikimania 2016 wiki too.

We already planned to keep a margin of around 20% considering that some authors won't make it. We have 42 papers accepted and 8 "accepted as reserve". If necessary, we will increase this reserve.

Yours,

Michele

On Wed, Feb 3, 2016 at 3:49 PM, Lodewijk <[hidden email]> wrote:
Hi Michele,

lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option. 

And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all. 

Lodewijk

On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza <[hidden email]> wrote:
Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much,
best regards,

Michele Lavazza
(Wikimania 2016 Programme Committee Deputy Chair)

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Lodewijk
Hi Michele,

posting on wiki is fine with me too - but not exactly what i had in mind (that would be a huge amount of copy & paste work). What I rather would suggest, is to use the Easychair built in functionality to share the feedback with the proposers. Please get in touch if you don't know how. 

Best,
Lodewijk

On Wed, Feb 3, 2016 at 7:12 PM, Michele Lavazza <[hidden email]> wrote:
Sure, we will upload the comments and feedback of pc members to the Wikimania 2016 wiki too.

We already planned to keep a margin of around 20% considering that some authors won't make it. We have 42 papers accepted and 8 "accepted as reserve". If necessary, we will increase this reserve.

Yours,

Michele

On Wed, Feb 3, 2016 at 3:49 PM, Lodewijk <[hidden email]> wrote:
Hi Michele,

lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option. 

And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all. 

Lodewijk

On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza <[hidden email]> wrote:
Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much,
best regards,

Michele Lavazza
(Wikimania 2016 Programme Committee Deputy Chair)

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Michele Lavazza
Sure, you're right. I will send the feedback to the authors immediately. That is indeed helpful for their understanding of the reasons for acceptance and, most importantly, rejection. I'm sorry I didn't do it before.

For the sake of transparency, however, the submissions and results will be published on the wiki too, even if you are right: that's quite an amount of work I'll share with Marco Chemello, so please be patient :)

Thank you very much for your help,
yours,

Michele

On Wed, Feb 3, 2016 at 7:27 PM, Lodewijk <[hidden email]> wrote:
Hi Michele,

posting on wiki is fine with me too - but not exactly what i had in mind (that would be a huge amount of copy & paste work). What I rather would suggest, is to use the Easychair built in functionality to share the feedback with the proposers. Please get in touch if you don't know how. 

Best,
Lodewijk

On Wed, Feb 3, 2016 at 7:12 PM, Michele Lavazza <[hidden email]> wrote:
Sure, we will upload the comments and feedback of pc members to the Wikimania 2016 wiki too.

We already planned to keep a margin of around 20% considering that some authors won't make it. We have 42 papers accepted and 8 "accepted as reserve". If necessary, we will increase this reserve.

Yours,

Michele

On Wed, Feb 3, 2016 at 3:49 PM, Lodewijk <[hidden email]> wrote:
Hi Michele,

lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option. 

And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all. 

Lodewijk

On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza <[hidden email]> wrote:
Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much,
best regards,

Michele Lavazza
(Wikimania 2016 Programme Committee Deputy Chair)

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Andy Mabbett-2

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Andy Mabbett-2

Here's another rejection-review, in full:

"Someone knows what wikispecies is about, and is worried about its future. Sounds like complaining to keep a project. No arguments were given. Topic to be discussed on mail lists."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Maarten Dammers
In reply to this post by Andy Mabbett-2
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Michele Lavazza
Dear Maarten,

I'm sorry that the process looks ridiculous to you. About the fact that only two people evaluated your submission, please take a look at https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation . I can only add that this year PC members, instead of evaluating all papers, had to bid for those they wanted to review. Therefore you have less reviews, but they are more qualified because your submission was chosen by someone who found it fell in his/her area of interest or of expertise.

About the fact that in your case, unfortunately, there are no verbal motivations for the numeric score, that depends on the fact that PC members were not obliged to give such motivations. Some did nevertheless.

About the fact that you've been sent the reviews without the names of the reviewers associated to them, I must blame the limitations of the EasyChair sofware, which from some points of view has been infernal to deal with. I'm sorry, as soon as we upload the submissions to the wiki you will see this information, too.

I hope I answered your questions.
Best regards,

Michele

On Wed, Feb 3, 2016 at 11:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Dariusz Jemielniak-3
In reply to this post by Maarten Dammers
hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Gnangarra
In reply to this post by Andy Mabbett-2
I would have thought knowing who was proposing the presentation was a significant factor in its evaluation as that provides the critical point of whether the person has the knowledge  to speak about the issue. Such games make the process appear to be unprofessional offering very little in the way of addressing critical issues and potentially opening up the community to abuse.  

Annon review is fine as that enables the reviewers to be critical without a concern of some backlash, I hope that before you publish the reviews publicly  that you consider the impact of those reviews, what is actually and ensure the reviews dont do harm to the person receiving them

On 4 February 2016 at 06:15, Andy Mabbett <[hidden email]> wrote:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--
GN.
President Wikimedia Australia
WMAU: http://www.wikimedia.org.au/wiki/User:Gnangarra
Photo Gallery: http://gnangarra.redbubble.com


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Phoebe Ayers-2
In reply to this post by Dariusz Jemielniak-3

Just to say that I agree with Dariusz's assessment of how academic reviewing is done. That doesn't address the tension of whether Wikimania is an academic conference or not, though - a question that has existed since early days. There's good arguments both ways, and each Wikimania team has approached this question differently.* It should, however, be made clear to submitters what is happening.

Phoebe

* many teams, and participants, have wanted a more academic conference for various reasons. Many participants find the socializing and practical knowledge sharing the most useful last. In recent years, we've tried to balance this by doing both. If I ever run Wikimania again, I think I'd try having *no* formal talks: only discussions and lightning talks.

On Feb 3, 2016 6:23 PM, "Dariusz Jemielniak" <[hidden email]> wrote:
hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Amir Sarabadani-2
Are they experts? in my review its "Interesting as it gives insight into how wiki data runs, wiki data is relatively unknown to most." (Even though this reviewer liked my submission but she/he gave me 6, two other ones gave me 9. Basically this reviewer changed status of my submission from accepted to "in reserve") This reviewer doesn't know the difference between "wikidata" and "wiki data" and she/he is reviewing my submission? Last year (and years before) reviewers were prominent contributors among users with huge knowledge around the projects. That's strange to me

Best

On Thu, Feb 4, 2016 at 3:13 AM phoebe ayers <[hidden email]> wrote:

Just to say that I agree with Dariusz's assessment of how academic reviewing is done. That doesn't address the tension of whether Wikimania is an academic conference or not, though - a question that has existed since early days. There's good arguments both ways, and each Wikimania team has approached this question differently.* It should, however, be made clear to submitters what is happening.

Phoebe

* many teams, and participants, have wanted a more academic conference for various reasons. Many participants find the socializing and practical knowledge sharing the most useful last. In recent years, we've tried to balance this by doing both. If I ever run Wikimania again, I think I'd try having *no* formal talks: only discussions and lightning talks.

On Feb 3, 2016 6:23 PM, "Dariusz Jemielniak" <[hidden email]> wrote:
hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

WereSpielChequers-2
In reply to this post by Dariusz Jemielniak-3
Two reviews per submission might work if we had a clear set of criteria that the reviewers were following and sufficient training of the reviewers that they were broadly consistent in their marking. But when you get the same presentation being marked as 5 and 8, as one of mine was then the suspicion is that the assessors are not working to the same criteria as each other. That wouldn't matter so much if they were all assessing all submissions, except that an assessor who varied between 0 and ten points would have far more influence than assessors who usually voted 6, 7 or 8. But having that level of inconsistency and only two reviews per submission makes the process a lottery that depends on who the two reviewers are for your submission.

As for the content of the reviews, I don't consider that either "5 (average)" or "6 (rather interesting) tell me anything as to why my submissions were rejected.

The other two reviews at least managed one or two lines. One of them even stretched to two sentences.

Hope Montreal manages something a bit better, I'm sure either Manilla or Perth would have done.


WereSpielChequers


On 3 Feb 2016, at 23:22, Dariusz Jemielniak <[hidden email]> wrote:

hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje
_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Federico Leva (Nemo)
In reply to this post by Lodewijk
Lodewijk, 03/02/2016 15:49:
> lots of people were quite surprised by the outcomes of the review.

This sounds like a good thing! If reviews matched expectations, there
would be no point in having a review. :)

Nemo

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Iolanda Pensa
In reply to this post by WereSpielChequers-2
dear all
to collect strengths and weakness of this system of review you can add you comments in the discussion page https://wikimania2016.wikimedia.org/wiki/Talk:Critical_issues_presentations#Feedback_and_Evaluation 
I have stated reporting your feedback, but please do not hesitate to correct, modify, add.

please consider
1. there are other kind of submissions: go for them! (we are updating them Monday February 8th) https://wikimania2016.wikimedia.org/wiki/Submissions
2. we are experimenting, surely to make Montreal better than us :) this is a clear objective of Wikimania Esino Lario https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Evaluation
3. thank you to all reviewers and people who made submissions!!! 

iolanda/iopensa


Il giorno 04 feb 2016, alle ore 09:22, WereSpielChequers <[hidden email]> ha scritto:

Two reviews per submission might work if we had a clear set of criteria that the reviewers were following and sufficient training of the reviewers that they were broadly consistent in their marking. But when you get the same presentation being marked as 5 and 8, as one of mine was then the suspicion is that the assessors are not working to the same criteria as each other. That wouldn't matter so much if they were all assessing all submissions, except that an assessor who varied between 0 and ten points would have far more influence than assessors who usually voted 6, 7 or 8. But having that level of inconsistency and only two reviews per submission makes the process a lottery that depends on who the two reviewers are for your submission.

As for the content of the reviews, I don't consider that either "5 (average)" or "6 (rather interesting) tell me anything as to why my submissions were rejected.

The other two reviews at least managed one or two lines. One of them even stretched to two sentences.

Hope Montreal manages something a bit better, I'm sure either Manilla or Perth would have done.


WereSpielChequers


On 3 Feb 2016, at 23:22, Dariusz Jemielniak <[hidden email]> wrote:

hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje
_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Tomasz Ganicz
In reply to this post by Dariusz Jemielniak-3
Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is  easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-)

Also  judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here:

https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation

Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were:

"
  1. problems and possible solutions in a specific field
  2. proposals for others to replicate
  3. issues (positive or negative) which have emerged from projects
  4. issues you want to raise which you feel have not been discussed yet
  5. issues which are at the centre of an online debate that you would like to address offline

"

1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre"  - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-)



2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak <[hidden email]>:
hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Federico Leva (Nemo)
Tomasz Ganicz, 04/02/2016 10:54:
> 1-4 are IMHO relatively easy to evaluate - I would expect from the
> reviews to answer yes or no to them. 5 is a bit tricky as it depends
> strongly of what the reviewer think is "at the centre"  - but I would
> expect that they at least explain in few words here what they think is
> "at the centre" or not :-)

Yet, reviews like the one Andy mentioned clearly address this: his
review was a clear "no" for point 5 as it pointed to (the absence of) a
mailing list discussion. (I can't check whether that's true.)

Nemo

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Tomasz Ganicz
In reply to this post by Tomasz Ganicz
Also - normally in Academia - if there are two strongly opposite reviews (one very positive, one very negative) a typical procedure is to send the submission to the third one.

2016-02-04 10:54 GMT+01:00 Tomasz Ganicz <[hidden email]>:
Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is  easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-)

Also  judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here:

https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation

Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were:

"
  1. problems and possible solutions in a specific field
  2. proposals for others to replicate
  3. issues (positive or negative) which have emerged from projects
  4. issues you want to raise which you feel have not been discussed yet
  5. issues which are at the centre of an online debate that you would like to address offline

"

1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre"  - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-)



2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak <[hidden email]>:
hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). 

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <[hidden email]> wrote:
What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW -----------
8


----------------------- REVIEW 2 ---------------------
PAPER: 194
TITLE: GLAM+Wikidata
AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW -----------
6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

--
Andy Mabbett
@pigsonthewing
http://pigsonthewing.org.uk



_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--

__________________________
prof. dr hab. Dariusz Jemielniak
kierownik katedry Zarządzania Międzynarodowego
i grupy badawczej NeRDS
Akademia Leona Koźmińskiego
http://nwrds.kozminski.edu.pl 

członek Akademii Młodych Uczonych Polskiej Akademii Nauk
członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l




--



--

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

Tomasz Ganicz
In reply to this post by Federico Leva (Nemo)
It is funny that my submission was rejected on the opposite POV as according to the reviewer:

"This is a community conference. There are plenty of opportunities for chapters, WMF, and politicians to get together. Esino Lario is not one of them."

I don't know why politicians were mentioned by review as the submission was about education programs of chapters and WMF, but anyway - yes it was addressed to Wikimedia educators - which of course can meet somewhere else, not necessarily in Esino Laro.

So - if you address your submission to the Wikimedia community - the issue is that you can contact community on this list for example, so it is good reason to reject, but if you address this to the more specific group they tell you that WIkimania is a meeting for "community" :-)

Taking such statements seriously - you can always say that you can meet anyone somewhere else to discuss the subject with them.

Also - this kind of  the reviewer statements do not correspond to what was written on criteria page:

"Please note that your presentation does not need to be for everyone; you can specify the target you would like to address."




2016-02-04 11:24 GMT+01:00 Federico Leva (Nemo) <[hidden email]>:
Tomasz Ganicz, 04/02/2016 10:54:
1-4 are IMHO relatively easy to evaluate - I would expect from the
reviews to answer yes or no to them. 5 is a bit tricky as it depends
strongly of what the reviewer think is "at the centre"  - but I would
expect that they at least explain in few words here what they think is
"at the centre" or not :-)

Yet, reviews like the one Andy mentioned clearly address this: his review was a clear "no" for point 5 as it pointed to (the absence of) a mailing list discussion. (I can't check whether that's true.)

Nemo


_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



--

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l
12