Worst. Survey. Ever.

classic Classic list List threaded Threaded
47 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Worst. Survey. Ever.

Steve Summit
Anybody know where on-wiki the current survey is being discussed?
I've got a thing or two to say.  (Message I just sent to
[hidden email] appended.)

                        * * *

From: Steve Summit <[hidden email]>
Date: Sat, 01 Nov 2008 10:16:41 -0400
To: [hidden email]
Subject: your survey has problems

I just completed the survey at http://survey47.wikipediastudy.org/
survey.php.  I'm sorry to be harsh and blunt.  It's terrible.
You can't use my results accurately -- they're wrong.
I doubt you can use anyone's results accurately.

This survey could only be completed accurately by someone:
* with nothing to do / too much time on their hands
* who never makes mistakes
* who can anticipate future questions before they're asked
* who can be bothered to search for his country and language
  (several times) in strictly-alphabetical lists of every single
  country and language in the world
* who knows the 2-character ISO code for the languages he knows,
  even when they're not obvious (e.g. DE for German)
* who knows the 3-character ISO code for the currency he uses

The survey told me I couldn't use my browser's Back and Forward
buttons, but had to use its own.  That's rude.

The survey then failed to provide Back buttons on all pages.
That's incompetent.

The survey then asked me questions like "How many hours do
you spend contributing to Wikipedia, per week?", followed by
"How many hours to you spend administering Wikipedia?", followed by
"How many hours do you spend supporting Wikipedia in technical ways?"
And that ended up being profoundly insulting.  Here's why.

The administrative and technical work I do on Wikipedia feels
like "contributions" to me, so (not knowing the next questions
were coming up) I included those hours in my first answer.
And the technical work I do feels like "administration", so
(not knowing the next question was coming up) I included that
in my second answer.  Therefore, if (as I suspect) you're
assuming those three categories are disjoint, and since my major
contributions lately have all been technical, I've inadvertently
overstated my overall contributions in this survey by a factor
of three.

And those particular survey pages were among those without
Back buttons, so I couldn't fix my mistake.  Do you know how
incredibly frustrating that is, to have wanted to spend time
contributing to a survey, to know I've contributed false
information, and to not be able to fix it?

Also, the survey took *way* too long.  And there was no
information given up-front about how long it might take.
The progress bar in the upper right-hand corner was a clue
and a nice touch, but it came too late.

The survey also took too long in relationship to the impression
of the data likely to be gleaned from it.  Short, tightly-focused
surveys give the surveyee the impression that some well-thought-out,
concise questions are being addressed by the surveyer.  Long,
scattershot surveys give the impression that the surveyers aren't
quite sure what they're looking for, are trying to ask everything
they can think of, and are imagining that they'll mine the data
later for interesting results later.  But, with poorly-defined
surveys, that task often ends up being difficult or impossible.
So I'm left begrudging the time I spent filling out the survey,
because it feels like the ratio of time investment (by me) to
useful information which can be gleaned (by you) is not good.

The survey asked me to specify things like "approximate number of
articles edited" and "percentage of time spent translating" using
drop-down selection boxes -- and with an increment of 1 between
the available choices!  That's just silly.  (I dreaded how long I
was going to have to scroll down to find my article edit count --
1196 -- and was both relieved and annoyed to discover that, after
500 entries, the drop-down list ended with "more than 500".)

The survey's categories were too-bluntly taken from existing
lists.  For example, the list I had to choose my employment from
was apparently taken from one of those dreadful Department of
Commerce categorizations, that I have just as much trouble
finding my job in when I fill out my tax forms.

At the very end, the survey asked if I wanted to submit my
results, or fix any mistakes.  But the provided way to fix
mistakes was to use the Back button -- perhaps several dozen
times -- which I wouldn't have felt like doing even if the chain
of Back buttons were complete.

The survey was clearly designed by someone who was thinking about
the data they wanted to collect, and in a scattershot way.  The
survey was clearly not designed with the person completing it in
mind.  The survey was clearly not designed or vetted by anyone
who knew anything about designing good surveys.

I probably had more complaints to list, but I shouldn't waste as
much time on this letter as I already wasted taking the survey,
so I'll stop here.

Bottom line: Please use the results of this survey with extreme
care, if at all.  The results are going to be heavily, heavily
biased by the inadvertent selection criteria involved in the
survey's hostility towards its participants.  If you conduct a
survey like this again, please find someone to assist in the
process who knows something about real-world survey work.

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Al Tally
On Sat, Nov 1, 2008 at 2:21 PM, Steve Summit <[hidden email]> wrote:

> Anybody know where on-wiki the current survey is being discussed?
> I've got a thing or two to say.  (Message I just sent to
> [hidden email] appended.)
>
>                        *       *       *
>

I haven't done the survey, and don't intend to. I had a brief look at it,
and decided not to bother when it said it would take half an hour. That's
half an hour that I could be spending working on an article. I also heard
surveys could be submitted more than once for one person, so effectively,
makes it worthless.

--
Alex
(User:Majorly)
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Giacomo M-Z
Afraid I gave up too after 5 minutes. Waste of time no-one ever tells the
truth in surveys anyway. Everyone always wants to appear more sophisticated
and clever than they really are.

Giano

On Sat, Nov 1, 2008 at 2:36 PM, Al Tally <[hidden email]>wrote:

> On Sat, Nov 1, 2008 at 2:21 PM, Steve Summit <[hidden email]> wrote:
>
> > Anybody know where on-wiki the current survey is being discussed?
> > I've got a thing or two to say.  (Message I just sent to
> > [hidden email] appended.)
> >
> >                        *       *       *
> >
>
> I haven't done the survey, and don't intend to. I had a brief look at it,
> and decided not to bother when it said it would take half an hour. That's
> half an hour that I could be spending working on an article. I also heard
> surveys could be submitted more than once for one person, so effectively,
> makes it worthless.
>
> --
> Alex
> (User:Majorly)
>  _______________________________________________
> WikiEN-l mailing list
> [hidden email]
> To unsubscribe from this mailing list, visit:
> https://lists.wikimedia.org/mailman/listinfo/wikien-l
>
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Nathan Awrich
Three points - the survey took me 10 minutes to complete, I didn't happen to
notice missing back buttons, and I don't think the Wikimedia Foundation (or
the English Wikipedia, as implied by where the complaints were posted)
designed the survey in detail. If you notice, it is an UNU-MERIT survey.
UNU-MERIT is a research institute of the United Nations University and
Maastricht University.

Nathan
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Erik Moeller-4
In reply to this post by Steve Summit
I agree that the "unanticipated follow-ups" are a serious design
problem that will distort some of the results, and in general, that
the technical implementation leaves lots to be desired, especially in
terms of usability. That said, there are many questions in the survey
which will yield useful data, or data that is useful with caveats. See
my summary here:
http://en.wikipedia.org/wiki/Wikipedia_talk:Survey_2008#Some_interim_comments
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Marc Riddell
In reply to this post by Nathan Awrich
on 11/1/08 11:16 AM, Nathan at [hidden email] wrote:

> Three points - the survey took me 10 minutes to complete, I didn't happen to
> notice missing back buttons, and I don't think the Wikimedia Foundation (or
> the English Wikipedia, as implied by where the complaints were posted)
> designed the survey in detail. If you notice, it is an UNU-MERIT survey.
> UNU-MERIT is a research institute of the United Nations University and
> Maastricht University.
>
Nathan, the survey is a disgrace in both form and content. And it is not
worthy of anyone's time or effort.

Marc Riddell


_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Thomas Dalton
In reply to this post by Steve Summit
> I just completed the survey at http://survey47.wikipediastudy.org/
> survey.php.  I'm sorry to be harsh and blunt.  It's terrible.
> You can't use my results accurately -- they're wrong.
> I doubt you can use anyone's results accurately.

I don't know if I'd go that far, but it's certainly not good.

> This survey could only be completed accurately by someone:
> * with nothing to do / too much time on their hands
True

> * who never makes mistakes
True

> * who can anticipate future questions before they're asked
That depends on how they intend to interpret the questions. That that
wasn't immeadiately clear is a serious problem

> * who can be bothered to search for his country and language
>  (several times) in strictly-alphabetical lists of every single
>  country and language in the world
Indeed - how difficult is it to put the most common languages at the
top? I expect at least 90% of respondents were from the top 10
languages.

> * who knows the 2-character ISO code for the languages he knows,
>  even when they're not obvious (e.g. DE for German)
How is DE not obvious? It's the first two letters of the language's
name in that language...

> * who knows the 3-character ISO code for the currency he uses

I struggled to find my currency (even knowing the code), it took me a
while to work out what they were actually sorting by.

> The survey told me I couldn't use my browser's Back and Forward
> buttons, but had to use its own.  That's rude.

That's a technical issue - it's certainly possible to do it in such a
way that back and forward buttons work, but not as easy.

> The survey then failed to provide Back buttons on all pages.
> That's incompetent.

True

> The survey then asked me questions like "How many hours do
> you spend contributing to Wikipedia, per week?", followed by
> "How many hours to you spend administering Wikipedia?", followed by
> "How many hours do you spend supporting Wikipedia in technical ways?"
> And that ended up being profoundly insulting.  Here's why.
>
> The administrative and technical work I do on Wikipedia feels
> like "contributions" to me, so (not knowing the next questions
> were coming up) I included those hours in my first answer.
> And the technical work I do feels like "administration", so
> (not knowing the next question was coming up) I included that
> in my second answer.  Therefore, if (as I suspect) you're
> assuming those three categories are disjoint, and since my major
> contributions lately have all been technical, I've inadvertently
> overstated my overall contributions in this survey by a factor
> of three.

I assumed they were intended the first question to be a total and so
answered the same as you. If that assumption was incorrect then my
response is also overstated.

> Also, the survey took *way* too long.  And there was no
> information given up-front about how long it might take.
> The progress bar in the upper right-hand corner was a clue
> and a nice touch, but it came too late.

Absolutely. I did finish it, but only because I'd got so far through
before realising how long it was taking. When it said it could take 30
mins to complete (of whatever it said), I assumed it was giving an
absolute maximum and it would actually be far shorter - it wasn't.

> The survey also took too long in relationship to the impression
> of the data likely to be gleaned from it.  Short, tightly-focused
> surveys give the surveyee the impression that some well-thought-out,
> concise questions are being addressed by the surveyer.  Long,
> scattershot surveys give the impression that the surveyers aren't
> quite sure what they're looking for, are trying to ask everything
> they can think of, and are imagining that they'll mine the data
> later for interesting results later.  But, with poorly-defined
> surveys, that task often ends up being difficult or impossible.
> So I'm left begrudging the time I spent filling out the survey,
> because it feels like the ratio of time investment (by me) to
> useful information which can be gleaned (by you) is not good.

Indeed - the first thing you need to work out when writing a survey is
what you want to learn from it. I'm not sure they did that...

> The survey asked me to specify things like "approximate number of
> articles edited" and "percentage of time spent translating" using
> drop-down selection boxes -- and with an increment of 1 between
> the available choices!  That's just silly.  (I dreaded how long I
> was going to have to scroll down to find my article edit count --
> 1196 -- and was both relieved and annoyed to discover that, after
> 500 entries, the drop-down list ended with "more than 500".)

I have no idea how many articles I've edited and guessed. I imagine
most other people guessed as well, which means having the numbers
accurate to 1 article is meaningless. They should have had groups
(0-10, 11-50, 51-100, 101-200, etc), the data would be just as useful
and it would be far quicker to fill out.

> The survey's categories were too-bluntly taken from existing
> lists.  For example, the list I had to choose my employment from
> was apparently taken from one of those dreadful Department of
> Commerce categorizations, that I have just as much trouble
> finding my job in when I fill out my tax forms.

It was the attempt to categorise what kind of articles you edit that
annoyed me. What does "General information" mean?

> At the very end, the survey asked if I wanted to submit my
> results, or fix any mistakes.  But the provided way to fix
> mistakes was to use the Back button -- perhaps several dozen
> times -- which I wouldn't have felt like doing even if the chain
> of Back buttons were complete.

A list of questions (without responses, since that would take up far
too much space) would have been good.

> The survey was clearly designed by someone who was thinking about
> the data they wanted to collect, and in a scattershot way.  The
> survey was clearly not designed with the person completing it in
> mind.  The survey was clearly not designed or vetted by anyone
> who knew anything about designing good surveys.

You don't need someone that's good at designing surveys (well you do,
but not to spot most of these problems), you just need to try the
survey out on a few people first.

> I probably had more complaints to list, but I shouldn't waste as
> much time on this letter as I already wasted taking the survey,
> so I'll stop here.
>
> Bottom line: Please use the results of this survey with extreme
> care, if at all.  The results are going to be heavily, heavily
> biased by the inadvertent selection criteria involved in the
> survey's hostility towards its participants.  If you conduct a
> survey like this again, please find someone to assist in the
> process who knows something about real-world survey work.

I was under the impression it was done with the support of experts -
if that's the case, pick better experts next time!

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Erik Moeller-4
2008/11/1 Thomas Dalton <[hidden email]>:
> You don't need someone that's good at designing surveys (well you do,
> but not to spot most of these problems), you just need to try the
> survey out on a few people first.

The survey was tried out on a group of testers and translators. You
only get so much useful feedback - the feedback that we're getting
from actually running the survey is much more detailed and valuable
for future surveys.

> I was under the impression it was done with the support of experts -
> if that's the case, pick better experts next time!

It was developed by the UNU-Merit Collaborative Creativity Group, who
have developed and run in-depth, multilingual surveys on the free
software movement, probably one of the most comparable specialized
communities. It's a first run, and the results will be imperfect and
need to be interpreted very carefully -- but we'll get some basic,
useful data, and we have a huge amount of feedback that will help with
the design of future surveys. I don't think we could have done much
better, especially given that the only resources we spent on this
project are staff time to shepherd it.
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Judson Dunn-2
On Sat, Nov 1, 2008 at 10:45 AM, Erik Moeller <[hidden email]> wrote:
>
> I don't think we could have done much
> better, especially given that the only resources we spent on this
> project are staff time to shepherd it.
> --

It's probably fine as a university study that Wikimedia is helping
with. Since we didn't spend much (if anything) on it, I wouldn't be
too hard on it. If one question has good results that may be worth it.
Even if we only get a relative ratio of people willing to take surveys
or something.

Having said that, I'm not going to take it, and I actually quite enjoy
taking and creating surveys, but this one sounds terrible from Steve's
description. I wonder if we can get the data about how many people
clicked on the take the survey link. That might be the only actually
good data available. (which I still consider a success, since the
foundation didn't actually spend much resources on it)

Judson
http://en.wikipedia.org/wiki/User:Cohesion

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Steve Summit
Judson Dunn wrote:
>On Sat, Nov 1, 2008 at 10:45 AM, Erik Moeller <[hidden email]> wrote:
>> I don't think we could have done much
>> better, especially given that the only resources we spent on this
>> project are staff time to shepherd it.
>
> It's probably fine as a university study that Wikimedia is helping
> with. Since we didn't spend much (if anything) on it, I wouldn't be
> too hard on it.

I wouldn't have been so hard on it, except that I was led to it
from a link appearing at the top of every Wikipedia page, a spot
that's usually reserved for things that are really significant
and really important.

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Al Tally
On Sat, Nov 1, 2008 at 4:09 PM, Steve Summit <[hidden email]> wrote:
>
> I wouldn't have been so hard on it, except that I was led to it
> from a link appearing at the top of every Wikipedia page, a spot
> that's usually reserved for things that are really significant
> and really important.


This link was added at the top of every project afaik, by Cary Bass, without
any discussion with the project's active editors, and no edit summary was
even used.

--
Alex
(User:Majorly)
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Erik Moeller-4
In reply to this post by Judson Dunn-2
2008/11/1 Judson Dunn <[hidden email]>:
> It's probably fine as a university study that Wikimedia is helping
> with. Since we didn't spend much (if anything) on it, I wouldn't be
> too hard on it. If one question has good results that may be worth it.
> Even if we only get a relative ratio of people willing to take surveys
> or something.

Lol, I would expect some more useful data than that. So far the
statistics indicate that there are almost 80,000 submitted
questionnaires, out of a total of 130,000 who at least took the first
question. That's a pretty high submission rate.

Looking again through the questionnaire, here are some of the
questions which I think will yield useful data:

* basic composition of sample (readers vs. contributors)
* basic demographics (gender, age, nationality, language, education level, etc.)
- exception: the "years of formal education" question will probably be
of limited usefulness; the occupation breakdown will have some gaps
* what contributors do
- exceptions: the detailed hours breakdown and category breakdown will
probably be of limited usefulness
* the "why contribute" reasons
* why non-contributors stopped contributing
* the "what purpose" question for readers
* the quality questions for readers
* the project and organization awareness questions

Sure, virtually every multiple choice question could have benefited
from additional choices, but that's always going to be the case -- you
can either try to process thousands of write-ins, or live with the
fact that some reasons will not be represented.

In general, there are some "numbers" questions which are dubious, but
we'll see what kind of data we get from those.

We won't get a representative selection of readers, but we wouldn't
get that anyway through a sitenotice survey. It's possible to just
interpret subsamples of the data which you want to examine to
understand e.g. differences between casual readers and frequent
readers.

The anonymized data will be CC-BY, so we'll all be able to get out of
it what's useful, and flag what's not.
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Nathan Awrich
What type of data substantiation are they planning on doing using the
username they ask for? A lot of the questions are moot once they have the
username - you can just look up simple data points like number of articles
edited etc.

Nathan
_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Erik Moeller-4
2008/11/1 Nathan <[hidden email]>:
> What type of data substantiation are they planning on doing using the
> username they ask for? A lot of the questions are moot once they have the
> username - you can just look up simple data points like number of articles
> edited etc.

Yes, for users who provide the name, they're planning to validate at
least the basic edit counts and such -- I'm not sure what additional
validation, but you can ask them at info(at)wikipediastudy(dot)org.

--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Michael Bimmler
In reply to this post by Steve Summit
I tend to agree with many of your comments on your survey and would
just like to pick some of the points I disagree with:

On Sat, Nov 1, 2008 at 3:21 PM, Steve Summit <[hidden email]> wrote:

> * who can be bothered to search for his country and language
>  (several times) in strictly-alphabetical lists of every single
>  country and language in the world

Well, one the one hand I am quite happy to have a list where it
doesn't say "United States", "United Kingdom" at the top, then two
dashes, and then all the "less important countries".

What I agree, though, is that this could have been made more
language-specific, if there had been more preparation time. E.g., if
someone chooses the German version of the survey, Germany, Austria and
Switzerland could have been at the top etc.

> * who knows the 2-character ISO code for the languages he knows,
>  even when they're not obvious (e.g. DE for German)

See, I think you should choose an example you know about next time ;-)
All German URLs end in .de, the German Wikipedia says
"de.wikipedia.org" etc.etc.
"DE" for German might not be obvious to a English-speaker, but it is
obvious to a German speaker and that's the whole point of it.

> * who knows the 3-character ISO code for the currency he uses
>

Come on, every bank statement of yours will tell you the ISO code of
the currency your account is in, you will probably find it on every
magazine that you read and so on and so on. Please don't tell me that
this is such an academic thing...

But these are minor points, again, I agree with the general direction
of your argument.

Michael

--
Michael Bimmler
[hidden email]

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Erik Moeller-4
2008/11/1 Michael Bimmler <[hidden email]>:
> Well, one the one hand I am quite happy to have a list where it
> doesn't say "United States", "United Kingdom" at the top, then two
> dashes, and then all the "less important countries".
>
> What I agree, though, is that this could have been made more
> language-specific, if there had been more preparation time. E.g., if
> someone chooses the German version of the survey, Germany, Austria and
> Switzerland could have been at the top etc.

Yep, I agree - this was one of our requests, but it was not to be.
These are usability quirks that make it more cumbersome than necessary
to complete the survey, but other than potentially more people not
answering these questions, should not influence the results.
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Gregory Maxwell
In reply to this post by Steve Summit
On Sat, Nov 1, 2008 at 10:21 AM, Steve Summit <[hidden email]> wrote:

> This survey could only be completed accurately by someone:
> * with nothing to do / too much time on their hands
> * who never makes mistakes
> * who can anticipate future questions before they're asked
> * who can be bothered to search for his country and language
>  (several times) in strictly-alphabetical lists of every single
>  country and language in the world
> * who knows the 2-character ISO code for the languages he knows,
>  even when they're not obvious (e.g. DE for German)
> * who knows the 3-character ISO code for the currency he uses
[snip]

While I would not use your harsh language, I did encounter many of the
same frustrations you did.


Un-anticipated follow ups made me re-consider my answers prior
questions, which I couldn't change without a back button. (For
example, my 'time spent' allocations will look screwy, because I
binned things together which shouldn't have been).

There were questions which I couldn't realistically provide precise
and reliable answers to such as "How many unique articles have you
started", "How many unique articles have you edited",  ... thought at
least it didn't expect me to provide answers with 1 unit granularity
for over 500.   (I still wasted a lot of time actually looking up the
correct answers, though I'm sure almost no one else would, and I ended
up having 'over 500' anyways).

There were several cases where I was frustrated by the answer I would
have ranked highest being unavailable. For example, many of my content
contributions to English Wikipedia are photographs. But that was never
an offered option, though write-ins helped.

It allowed you to tell it about contributions in multiple projects and
languages, but didn't really provide a facility to express that your
contributions were different in different languages.   (For example,
being an admin on some projects and not others will result in vastly
different distributions of time).

I hope the question tree for people who are only readers is somewhat better.

In the future it might be helpful if the questions were made available
in advance to more than just translators. I specifically tried to find
the question sheet in advance on this one. I doubt I would have caught
the unanticipated followups without actually taking it, but I would
have pointed out a couple things which could have improved.

All that said I don't share Steve's pessimism: I expect the results of
the survey to be interesting regardless of the survey's shortcomings.

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Thomas Dalton
In reply to this post by Erik Moeller-4
2008/11/1 Erik Moeller <[hidden email]>:

> 2008/11/1 Michael Bimmler <[hidden email]>:
>> Well, one the one hand I am quite happy to have a list where it
>> doesn't say "United States", "United Kingdom" at the top, then two
>> dashes, and then all the "less important countries".
>>
>> What I agree, though, is that this could have been made more
>> language-specific, if there had been more preparation time. E.g., if
>> someone chooses the German version of the survey, Germany, Austria and
>> Switzerland could have been at the top etc.
>
> Yep, I agree - this was one of our requests, but it was not to be.
> These are usability quirks that make it more cumbersome than necessary
> to complete the survey, but other than potentially more people not
> answering these questions, should not influence the results.

Well, you may end up with a disproportionate representation of people
speaking languages near the beginning of the alphabet. The data about
what speakers of each language do should be fine, but the data about
how many people speak each language will be unreliable.

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Erik Moeller-4
2008/11/1 Thomas Dalton <[hidden email]>:
> Well, you may end up with a disproportionate representation of people
> speaking languages near the beginning of the alphabet.

Putting some countries first protects against such selection bias in
common cases, though it could potentially introduce other biases
(countries not in the top list may be underrepresented). The only way
to truly protect against selection biases of any kind is to randomize
the list, which obviously is much more cumbersome.

We'll have to see the actual data to assess how large these potential
distortions might be. For example, if 95% of respondents completed the
country/languages questions, then the selection bias of not finding
your country is probably relatively small.
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
Reply | Threaded
Open this post in threaded view
|

Re: Worst. Survey. Ever.

Thomas Dalton
2008/11/1 Erik Moeller <[hidden email]>:
> 2008/11/1 Thomas Dalton <[hidden email]>:
>> Well, you may end up with a disproportionate representation of people
>> speaking languages near the beginning of the alphabet.
>
> Putting some countries first protects against such selection bias in
> common cases, though it could potentially introduce other biases
> (countries not in the top list may be underrepresented). The only way
> to truly protect against selection biases of any kind is to randomize
> the list, which obviously is much more cumbersome.

Indeed, there is no ideal solution.

> We'll have to see the actual data to assess how large these potential
> distortions might be. For example, if 95% of respondents completed the
> country/languages questions, then the selection bias of not finding
> your country is probably relatively small.

It's the people that stopped answering questions completely just
before the language questions that are the problem - there is no way
to know if they gave up because they couldn't find their language or
because they'd just had enough. Obviously, if very few people stopped
at that point then it doesn't matter, but chances are a significant
number would have stopped at that point by random chance which makes
it difficult to interpret the data.

_______________________________________________
WikiEN-l mailing list
[hidden email]
To unsubscribe from this mailing list, visit:
https://lists.wikimedia.org/mailman/listinfo/wikien-l
123