Usability Study Results (Sneak Preview)

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Usability Study Results (Sneak Preview)

Parul Vora
Hi all!

The Wikipedia Usability Initiative conducted a user research study with
SF based Bolt Peters in late March to uncover barriers new editors face.
We are in the process of completing a full report on our methodology,
process and analysis, but wanted to share with you some of the major
themes and findings in the meantime....


Some quotes from our participants that illustrate these findings:

“Usually it’s the most information in the easiest spot to access. It
always looks very well put together….it boggles my mind how many people
can contribute and it still looks like an encyclopedia.” – ‘Galen’

“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’

“Rather than making a mess, I’d rather take some time to figure out how
to do it right."
(later) "There sure is a lot of stuff to read.” – ‘Dan’

“ [I felt] kind of stupid.” – ‘Galen’

“It’d be nice to have a GUI, so you could see what you’re editing.
You’ve made these changes and you’re looking at it, and you don’t know
how it’s going to look on the page. It’s a little clumsy to see how it’s
going to look.” – ‘Bryan’

“[This is] where I’d give up.” – ‘Shaun’


Check out the full post on the foundation blog:
http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/


We would love to hear any initial thoughts, opinions, and reactions. If
you have any similar or dissimilar experiences - either personally or in
your own work/research, we'd love to hear about that too!


Always on your side,
The Usability Team

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Magnus Manske-2
On Fri, Apr 24, 2009 at 2:08 AM, Parul Vora <[hidden email]> wrote:
> Hi all!
>
> The Wikipedia Usability Initiative conducted a user research study with
> SF based Bolt Peters in late March to uncover barriers new editors face.
> We are in the process of completing a full report on our methodology,
> process and analysis, but wanted to share with you some of the major
> themes and findings in the meantime....

From what I read, the main problem is that new, eager, serious
contributers surrender between our markup and an overwhelming flood of
descriptions.

I know a new GUI is being worked on. For the moment I hacked the
following JavaScript suggestion:

http://en.wikipedia.org/wiki/User:Magnus_Manske/newbiehelp.js

This adds a "how?" link into the "edit" tab, and launches a floating
panel with some extremely general content:

http://en.wikipedia.org/wiki/File:Edit_how.png

Never mind the wording, the color scheme, or important points I missed :-)

If that were added for all anons by default, it might save the willing
and able some grief.

Just a thought.

Cheers,
Magnus

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

thekohser
In reply to this post by Parul Vora
Will the final report include a note about how "unwelcome" User:NawlinWiki
made the study participants feel when he indefinitely blocked their accounts
for "abusing" Wikipedia?

http://en.wikipedia.org/w/index.php?title=Special:Log&page=User%3AUsability_Tester_3
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Naoko Komura
In reply to this post by Magnus Manske-2
>
>
> I know a new GUI is being worked on. For the moment I hacked the
> following JavaScript suggestion:
>
> http://en.wikipedia.org/wiki/User:Magnus_Manske/newbiehelp.js
>
> This adds a "how?" link into the "edit" tab, and launches a floating
> panel with some extremely general content:
>
> http://en.wikipedia.org/wiki/File:Edit_how.png
>
> Never mind the wording, the color scheme, or important points I missed :-)
>
> If that were added for all anons by default, it might save the willing
> and able some grief.
>
> Just a thought.
>
>
Great suggestion.  One of the repeated sentiment from the study participants
was "what is the editing process".  This will give a quick overview to
anons.

Will you post your idea to our project page?  :-)
http://usability.wikimedia.org/wiki/Main_Page


Best,

- Naoko

--
Support Free Knowledge:  http://wikimediafoundation.org/wiki/Donate
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Naoko Komura
In reply to this post by thekohser
On Fri, Apr 24, 2009 at 10:26 AM, Gregory Kohs <[hidden email]> wrote:

> Will the final report include a note about how "unwelcome" User:NawlinWiki
> made the study participants feel when he indefinitely blocked their
> accounts
> for "abusing" Wikipedia?


We, the usability team, with lots of help from stewards and admins, worked
behind the scene not to expose such blockage to study participants.  I am
sure NawlinkWiki was doing his part in protecting Wikipedia.  We notified
this list and WikiEn about the usability study and described the usage of
these accounts in the user page, hoping to avoid such blockage.  But some
folks are double-cautious and I think that is a good thing.  We confirmed
that it is really the usability team of WMF doing the test, and could clear
the concern.  So it was not a problem.

Best,

- Naoko





--
Support Free Knowledge:  http://wikimediafoundation.org/wiki/Donate
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Parul Vora
In reply to this post by Parul Vora
Hi All!

Thanks for all of the feedback, comments, and support. I just wanted to
let you know that our full report (including highlight videos!!) is now
up our the Usability Initiative's project wiki:

http://usability.wikimedia.org/wiki/UX_and_Usability_Study

- The Usability Team


Parul Vora wrote:

> Hi all!
>
> The Wikipedia Usability Initiative conducted a user research study
> with SF based Bolt Peters in late March to uncover barriers new
> editors face. We are in the process of completing a full report on our
> methodology, process and analysis, but wanted to share with you some
> of the major themes and findings in the meantime....
>
>
> Some quotes from our participants that illustrate these findings:
>
> “Usually it’s the most information in the easiest spot to access. It
> always looks very well put together….it boggles my mind how many
> people can contribute and it still looks like an encyclopedia.” – ‘Galen’
>
> “I like Wikipedia because it’s plain text and nothing flashes” –
> ‘Claudia’
>
> “Rather than making a mess, I’d rather take some time to figure out
> how to do it right."
> (later) "There sure is a lot of stuff to read.” – ‘Dan’
>
> “ [I felt] kind of stupid.” – ‘Galen’
>
> “It’d be nice to have a GUI, so you could see what you’re editing.
> You’ve made these changes and you’re looking at it, and you don’t know
> how it’s going to look on the page. It’s a little clumsy to see how
> it’s going to look.” – ‘Bryan’
>
> “[This is] where I’d give up.” – ‘Shaun’
>
>
> Check out the full post on the foundation blog:
> http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/ 
>
>
>
> We would love to hear any initial thoughts, opinions, and reactions.
> If you have any similar or dissimilar experiences - either personally
> or in your own work/research, we'd love to hear about that too!
>
>
> Always on your side,
> The Usability Team
>


_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
This usability study is so tiny. I want MediaWiki to be really, really good.
Please tell me you guys hope to go large scale with the remote testing
setup.

On Thu, May 7, 2009 at 5:41 PM, Parul Vora <[hidden email]> wrote:

> Hi All!
>
> Thanks for all of the feedback, comments, and support. I just wanted to
> let you know that our full report (including highlight videos!!) is now
> up our the Usability Initiative's project wiki:
>
> http://usability.wikimedia.org/wiki/UX_and_Usability_Study
>
> - The Usability Team
>
>
> Parul Vora wrote:
> > Hi all!
> >
> > The Wikipedia Usability Initiative conducted a user research study
> > with SF based Bolt Peters in late March to uncover barriers new
> > editors face. We are in the process of completing a full report on our
> > methodology, process and analysis, but wanted to share with you some
> > of the major themes and findings in the meantime....
> >
> >
> > Some quotes from our participants that illustrate these findings:
> >
> > “Usually it’s the most information in the easiest spot to access. It
> > always looks very well put together….it boggles my mind how many
> > people can contribute and it still looks like an encyclopedia.” – ‘Galen’
> >
> > “I like Wikipedia because it’s plain text and nothing flashes” –
> > ‘Claudia’
> >
> > “Rather than making a mess, I’d rather take some time to figure out
> > how to do it right."
> > (later) "There sure is a lot of stuff to read.” – ‘Dan’
> >
> > “ [I felt] kind of stupid.” – ‘Galen’
> >
> > “It’d be nice to have a GUI, so you could see what you’re editing.
> > You’ve made these changes and you’re looking at it, and you don’t know
> > how it’s going to look on the page. It’s a little clumsy to see how
> > it’s going to look.” – ‘Bryan’
> >
> > “[This is] where I’d give up.” – ‘Shaun’
> >
> >
> > Check out the full post on the foundation blog:
> >
> http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
> >
> >
> >
> > We would love to hear any initial thoughts, opinions, and reactions.
> > If you have any similar or dissimilar experiences - either personally
> > or in your own work/research, we'd love to hear about that too!
> >
> >
> > Always on your side,
> > The Usability Team
> >
>
>
> _______________________________________________
> foundation-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
>
> Based on these criteria, the 2,500 users that responded to our survey were
> filtered down to 500 viable subjects based on their answers to these
> questions. The team, along with B|P, partnered with Davis Recruiting to
> contact, filter, and screen these 500 participants based on their Wikipedia
> contribution history, Wikipedia usage patterns, their given reasons for not
> contributing, and their talkativeness and openness to discuss their thoughts
> and actions. From 2,500 users, we ended up with 10 study participants and
> 3-5 waitlisted participants.


You went from 2,500 subjects to just 10? Remote testing allows you to study
a virtually unlimited number of participants in a fully natural environment,
not some clean room. If you're going to clean up the interface by moving
high-utility elements to the areas users tend to look at most, then I'd
recommend bringing them into the office. Otherwise, you need to catch them
in their regular routine - they perform a Google search, it lands them at
Wikipedia. What happens next? Do you feel like you have the answer to this
question after your local studies? They see a bit of information is
incorrect. How many users *just don't get what Wikipedia is* at that stage?
Of those who get it, what happens next? I'm talking about in the course of
their day, in the middle of whatever it is they were doing they needed
information, how does Wikipedia actually get used? That's a usability test,
and you have no clue!



On Thu, May 7, 2009 at 6:22 PM, Brian <[hidden email]> wrote:

> This usability study is so tiny. I want MediaWiki to be really, really
> good. Please tell me you guys hope to go large scale with the remote testing
> setup.
>
>
> On Thu, May 7, 2009 at 5:41 PM, Parul Vora <[hidden email]> wrote:
>
>> Hi All!
>>
>> Thanks for all of the feedback, comments, and support. I just wanted to
>> let you know that our full report (including highlight videos!!) is now
>> up our the Usability Initiative's project wiki:
>>
>> http://usability.wikimedia.org/wiki/UX_and_Usability_Study
>>
>> - The Usability Team
>>
>>
>> Parul Vora wrote:
>> > Hi all!
>> >
>> > The Wikipedia Usability Initiative conducted a user research study
>> > with SF based Bolt Peters in late March to uncover barriers new
>> > editors face. We are in the process of completing a full report on our
>> > methodology, process and analysis, but wanted to share with you some
>> > of the major themes and findings in the meantime....
>> >
>> >
>> > Some quotes from our participants that illustrate these findings:
>> >
>> > “Usually it’s the most information in the easiest spot to access. It
>> > always looks very well put together….it boggles my mind how many
>> > people can contribute and it still looks like an encyclopedia.” –
>> ‘Galen’
>> >
>> > “I like Wikipedia because it’s plain text and nothing flashes” –
>> > ‘Claudia’
>> >
>> > “Rather than making a mess, I’d rather take some time to figure out
>> > how to do it right."
>> > (later) "There sure is a lot of stuff to read.” – ‘Dan’
>> >
>> > “ [I felt] kind of stupid.” – ‘Galen’
>> >
>> > “It’d be nice to have a GUI, so you could see what you’re editing.
>> > You’ve made these changes and you’re looking at it, and you don’t know
>> > how it’s going to look on the page. It’s a little clumsy to see how
>> > it’s going to look.” – ‘Bryan’
>> >
>> > “[This is] where I’d give up.” – ‘Shaun’
>> >
>> >
>> > Check out the full post on the foundation blog:
>> >
>> http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
>> >
>> >
>> >
>> > We would love to hear any initial thoughts, opinions, and reactions.
>> > If you have any similar or dissimilar experiences - either personally
>> > or in your own work/research, we'd love to hear about that too!
>> >
>> >
>> > Always on your side,
>> > The Usability Team
>> >
>>
>>
>> _______________________________________________
>> foundation-l mailing list
>> [hidden email]
>> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>>
>
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

geni
In reply to this post by Brian J Mingus
2009/5/8 Brian <[hidden email]>:
> This usability study is so tiny. I want MediaWiki to be really, really good.
> Please tell me you guys hope to go large scale with the remote testing
> setup.

Nit just mediawiki. Looks like we need to improve the paths through
the help namespace.

--
geni

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

David Gerard-2
In reply to this post by Brian J Mingus
2009/5/8 Brian <[hidden email]>:

> You went from 2,500 subjects to just 10?


For a software test, which this mostly was, 5 is enough for excellent
results in most cases.


- d.

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
In reply to this post by geni
That may be true. This study does not allow you to draw that conclusion,
however.

On Thu, May 7, 2009 at 6:36 PM, geni <[hidden email]> wrote:

> 2009/5/8 Brian <[hidden email]>:
> > This usability study is so tiny. I want MediaWiki to be really, really
> good.
> > Please tell me you guys hope to go large scale with the remote testing
> > setup.
>
> Nit just mediawiki. Looks like we need to improve the paths through
> the help namespace.
>
> --
> geni
>
> _______________________________________________
> foundation-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Erik Moeller-4
In reply to this post by Brian J Mingus
2009/5/7 Brian <[hidden email]>:
>>
>> Based on these criteria, the 2,500 users that responded to our survey were
>> filtered down to 500 viable subjects based on their answers to these
>> questions. The team, along with B|P, partnered with Davis Recruiting to
>> contact, filter, and screen these 500 participants based on their Wikipedia
>> contribution history, Wikipedia usage patterns, their given reasons for not
>> contributing, and their talkativeness and openness to discuss their thoughts
>> and actions. From 2,500 users, we ended up with 10 study participants and
>> 3-5 waitlisted participants.

> You went from 2,500 subjects to just 10?

The purpose of a study like this is focused observation of the
behavior of individual human beings. As David has pointed out, for any
study like this there are laws of diminishing returns, and any serious
observation of an individual is time-consuming and costly (raw data is
worthless if you can't analyze it). That's why usability gurus like
Nielsen suggest "5 is enough" for most tests:
http://www.useit.com/alertbox/20000319.html - due to our highly
diverse audience, we chose a larger group, and we split between remote
and lab testing to compensate for biases of both methods. This has
worked well to identify plenty of very obvious usability barriers to
focus on.

There are alternative data collection methods such as large scale
quantitative testing where the level of individual engagement is
limited; those can give you behavioral patterns etc. They can be
useful, too, but are an entirely different thing.
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
This all goes back to how you aim to quantify improvement in usability.
These samples sizes are so small that it will be hard (or even impossible)
to evaluate your progress based on statistical significance. You've got to
prove to us that its really getting better, and doesn't just look prettier.

On Thu, May 7, 2009 at 7:44 PM, Erik Moeller <[hidden email]> wrote:

> 2009/5/7 Brian <[hidden email]>:
> >>
> >> Based on these criteria, the 2,500 users that responded to our survey
> were
> >> filtered down to 500 viable subjects based on their answers to these
> >> questions. The team, along with B|P, partnered with Davis Recruiting to
> >> contact, filter, and screen these 500 participants based on their
> Wikipedia
> >> contribution history, Wikipedia usage patterns, their given reasons for
> not
> >> contributing, and their talkativeness and openness to discuss their
> thoughts
> >> and actions. From 2,500 users, we ended up with 10 study participants
> and
> >> 3-5 waitlisted participants.
>
> > You went from 2,500 subjects to just 10?
>
> The purpose of a study like this is focused observation of the
> behavior of individual human beings. As David has pointed out, for any
> study like this there are laws of diminishing returns, and any serious
> observation of an individual is time-consuming and costly (raw data is
> worthless if you can't analyze it). That's why usability gurus like
> Nielsen suggest "5 is enough" for most tests:
> http://www.useit.com/alertbox/20000319.html - due to our highly
> diverse audience, we chose a larger group, and we split between remote
> and lab testing to compensate for biases of both methods. This has
> worked well to identify plenty of very obvious usability barriers to
> focus on.
>
> There are alternative data collection methods such as large scale
> quantitative testing where the level of individual engagement is
> limited; those can give you behavioral patterns etc. They can be
> useful, too, but are an entirely different thing.
> --
> Erik Möller
> Deputy Director, Wikimedia Foundation
>
> Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
>
> _______________________________________________
> foundation-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Erik Moeller-4
2009/5/7 Brian <[hidden email]>:
> This all goes back to how you aim to quantify improvement in usability.
> These samples sizes are so small that it will be hard (or even impossible)
> to evaluate your progress based on statistical significance. You've got to
> prove to us that its really getting better, and doesn't just look prettier.

That's why, in addition to follow-up tests, the usability team is
working closely with Erik Zachte to make sure we've got good editing
metrics that show whether our changes are actually making a
substantial difference in engaging new contributors.
--
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
Quite frankly the advice that you should only use five subjects makes no
sense. The appeal to Nielsen's authority is not going to work on me or
anyone else who understands why the scientific method exists. It's
unscientific thinking and it's going cause to you waste money. You're going
to draw conclusions based on results that simply aren't valid, and you won't
know it until the study is over and you didn't make progress.

Careful analysis of site data could allow you to draw some conclusions. I'm
curious how you're planning to go about that. Dependent/independent
variables?

On Thu, May 7, 2009 at 7:54 PM, Erik Moeller <[hidden email]> wrote:

> 2009/5/7 Brian <[hidden email]>:
> > This all goes back to how you aim to quantify improvement in usability.
> > These samples sizes are so small that it will be hard (or even
> impossible)
> > to evaluate your progress based on statistical significance. You've got
> to
> > prove to us that its really getting better, and doesn't just look
> prettier.
>
> That's why, in addition to follow-up tests, the usability team is
> working closely with Erik Zachte to make sure we've got good editing
> metrics that show whether our changes are actually making a
> substantial difference in engaging new contributors.
> --
> Erik Möller
> Deputy Director, Wikimedia Foundation
>
> Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
>
> _______________________________________________
> foundation-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Robert Rohde
On Thu, May 7, 2009 at 8:58 PM, Brian <[hidden email]> wrote:

> Quite frankly the advice that you should only use five subjects makes no
> sense. The appeal to Nielsen's authority is not going to work on me or
> anyone else who understands why the scientific method exists. It's
> unscientific thinking and it's going cause to you waste money. You're going
> to draw conclusions based on results that simply aren't valid, and you won't
> know it until the study is over and you didn't make progress.
>
> Careful analysis of site data could allow you to draw some conclusions. I'm
> curious how you're planning to go about that. Dependent/independent
> variables?

If five subjects, chosen at random, all have the same problem, then
with 95% confidence you can predict that at least half of the
population will report having this problem.

This kind of work generally focuses on BIG problems, and you don't
need a huge sample to identify some of the most common issues.  In
things like UI development it would be surprising if there weren't
complaints reported by most of the subjects.  You may overlook some
other problems, but when coming up with a list of common problems to
work on, I would say that 15 subjects is plenty.

-Robert Rohde

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Sage Ross
In reply to this post by Brian J Mingus
On Thu, May 7, 2009 at 11:58 PM, Brian <[hidden email]> wrote:

> Quite frankly the advice that you should only use five subjects makes no
> sense. The appeal to Nielsen's authority is not going to work on me or
> anyone else who understands why the scientific method exists. It's
> unscientific thinking and it's going cause to you waste money. You're going
> to draw conclusions based on results that simply aren't valid, and you won't
> know it until the study is over and you didn't make progress.
>
> Careful analysis of site data could allow you to draw some conclusions. I'm
> curious how you're planning to go about that. Dependent/independent
> variables?
>

An exercise in statistical thinking: when everyone or almost everyone
cites problem X, how many people does it take to reach statistical
significance that X is a problem worth addressing?  Even if the
results are a statistical fluke and in reality only 20% of new users
run into trouble with problem X, that's still a problem worth
addressing.

The fact that so many of the 15 people had the same problems, and
those problems also align with common sense, is a strong indication
that the study has found some things worth fixing.

There is more than one way to come to reliable conclusions.  Any time
I see someone invoking "the [singular] scientific method", as if there
is only one and it is set in stone and universally agreed upon by all
rational people, I have trouble taking them seriously.  See
[[Talk:Scientific method]].

-Sage (User:Ragesoss)

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
I will have no part in your efforts to redefine the scientific method on its
talk page.

On Thu, May 7, 2009 at 10:35 PM, Sage Ross
<[hidden email]<ragesoss%[hidden email]>
> wrote:

> On Thu, May 7, 2009 at 11:58 PM, Brian <[hidden email]> wrote:
> > Quite frankly the advice that you should only use five subjects makes no
> > sense. The appeal to Nielsen's authority is not going to work on me or
> > anyone else who understands why the scientific method exists. It's
> > unscientific thinking and it's going cause to you waste money. You're
> going
> > to draw conclusions based on results that simply aren't valid, and you
> won't
> > know it until the study is over and you didn't make progress.
> >
> > Careful analysis of site data could allow you to draw some conclusions.
> I'm
> > curious how you're planning to go about that. Dependent/independent
> > variables?
> >
>
> An exercise in statistical thinking: when everyone or almost everyone
> cites problem X, how many people does it take to reach statistical
> significance that X is a problem worth addressing?  Even if the
> results are a statistical fluke and in reality only 20% of new users
> run into trouble with problem X, that's still a problem worth
> addressing.
>
> The fact that so many of the 15 people had the same problems, and
> those problems also align with common sense, is a strong indication
> that the study has found some things worth fixing.
>
> There is more than one way to come to reliable conclusions.  Any time
> I see someone invoking "the [singular] scientific method", as if there
> is only one and it is set in stone and universally agreed upon by all
> rational people, I have trouble taking them seriously.  See
> [[Talk:Scientific method]].
>
> -Sage (User:Ragesoss)
>
> _______________________________________________
> foundation-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Brian J Mingus
In reply to this post by Robert Rohde
Sounds easy. I wonder why this "study" doesn't mention a p value. The grant
must not have been large enough to fund someone with any experience using R,
or god forbid, a pencil.

On Thu, May 7, 2009 at 10:32 PM, Robert Rohde <[hidden email]> wrote:

> On Thu, May 7, 2009 at 8:58 PM, Brian <[hidden email]> wrote:
> > Quite frankly the advice that you should only use five subjects makes no
> > sense. The appeal to Nielsen's authority is not going to work on me or
> > anyone else who understands why the scientific method exists. It's
> > unscientific thinking and it's going cause to you waste money. You're
> going
> > to draw conclusions based on results that simply aren't valid, and you
> won't
> > know it until the study is over and you didn't make progress.
> >
> > Careful analysis of site data could allow you to draw some conclusions.
> I'm
> > curious how you're planning to go about that. Dependent/independent
> > variables?
>
> If five subjects, chosen at random, all have the same problem, then
> with 95% confidence you can predict that at least half of the
> population will report having this problem.
>
> This kind of work generally focuses on BIG problems, and you don't
> need a huge sample to identify some of the most common issues.  In
> things like UI development it would be surprising if there weren't
> complaints reported by most of the subjects.  You may overlook some
> other problems, but when coming up with a list of common problems to
> work on, I would say that 15 subjects is plenty.
>
> -Robert Rohde
>
> _______________________________________________
> foundation-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Reply | Threaded
Open this post in threaded view
|

Re: Usability Study Results (Sneak Preview)

Nikola Smolenski
In reply to this post by Parul Vora
Parul Vora wrote:
> Thanks for all of the feedback, comments, and support. I just wanted to
> let you know that our full report (including highlight videos!!) is now
> up our the Usability Initiative's project wiki:
>
> http://usability.wikimedia.org/wiki/UX_and_Usability_Study

It may be too late for this now, but when I read that "The cheat sheet
was the only item in the help section that led to a subjects successful
edit.", I wanted to know how well would people edit if they had the
printed cheat sheet. Maybe if you have more tests in future you could do
this - it will tell us exactly what is the value of the printed
promotional material.

I am wary of this: "Users often missed the ‘edit’ buttons next to each
section, clicking on ‘edit this page’ all the way at the top." In my
experience, users do exactly the opposite, and I have seen new users who
know how to edit sections asking how to edit top section; some
Wikipedias (f.e. ruwiki) have even added [edit] link to top of the
article that mimics section edit links. What could be the cause of this
discrepancy?

Would it be possible to get some conclusions and recommendations from
this study so that Wikipedia's interface could be improved even before a
full WYSIVYG interface is developed? (Because, from past experience,
that will take years.) For example, one obvious thing is that editing
help should be moved to top right of the edit box. (Maybe it could also
pop next to the editing box instead of in a new window?) Another obvious
thing, "Create a new article" link should be added somewhere. Yet
another, toolbar images should be replaced with text.

_______________________________________________
foundation-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
12