Browser tests for core

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Browser tests for core

Chris Steipp
I just +2'ed a change to add a few basic selenium tests to core [1]. I
think it will benefit us all to have a set of automated tests to
quickly make sure mediawiki is working correctly. From a security
perspective, this also takes a step towards more efficient security
testing, which I'm also a fan of (if you've tried blindly scanning
mediawiki, you know what I'm talking about..).

I think the QA group is working on vagrant-izing these, but if you
have ruby >1.9.3 and firefox, then setting up and running these tests
on your local dev system is 4-6 commands,

$ cd tests/browser
$ gem update --system
$ gem install bundler
$ bundle install

You can either set your environment variables yourself, or edit
environment_variables and run `source environment_variables` to set
them. Then it's just

$ bundle exec cucumber features/

to run the tests. They currently complete in 36 seconds on my laptop.

I'd like to see more tests added and backported to REL1_23 to make
sure we have an ongoing suite to check releases against for next few
years that we support that LTS. If anyone is interested in both
mediawiki core and browser tests, I'm sure the QA team would like to
get you involved.

Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
and getting this done. I'll let them jump in with all the details I've
missed.


[1] - https://gerrit.wikimedia.org/r/#/c/133507/

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

James Forrester-4
On 24 June 2014 15:46, Chris Steipp <[hidden email]> wrote:

> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly.


​Great news; thank you all!

J.
--
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

[hidden email] | @jdforrester
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Ryan Kaldari-2
In reply to this post by Chris Steipp
Nice work guys! One slight issue though. The user Selenium_user (as set in
environment_variables) was just indefinitely blocked on en.wiki :(

Ryan Kaldari


On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp <[hidden email]> wrote:

> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly. From a security
> perspective, this also takes a step towards more efficient security
> testing, which I'm also a fan of (if you've tried blindly scanning
> mediawiki, you know what I'm talking about..).
>
> I think the QA group is working on vagrant-izing these, but if you
> have ruby >1.9.3 and firefox, then setting up and running these tests
> on your local dev system is 4-6 commands,
>
> $ cd tests/browser
> $ gem update --system
> $ gem install bundler
> $ bundle install
>
> You can either set your environment variables yourself, or edit
> environment_variables and run `source environment_variables` to set
> them. Then it's just
>
> $ bundle exec cucumber features/
>
> to run the tests. They currently complete in 36 seconds on my laptop.
>
> I'd like to see more tests added and backported to REL1_23 to make
> sure we have an ongoing suite to check releases against for next few
> years that we support that LTS. If anyone is interested in both
> mediawiki core and browser tests, I'm sure the QA team would like to
> get you involved.
>
> Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> and getting this done. I'll let them jump in with all the details I've
> missed.
>
>
> [1] - https://gerrit.wikimedia.org/r/#/c/133507/
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Greg Grossmeier-2
<quote name="Ryan Kaldari" date="2014-06-24" time="15:56:31 -0700">
> Nice work guys! One slight issue though. The user Selenium_user (as set in
> environment_variables) was just indefinitely blocked on en.wiki :(

That shouldn't affect these tests. These run on your local box/vagrant.

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Chris McMahon
In reply to this post by Ryan Kaldari-2
On Tue, Jun 24, 2014 at 3:56 PM, Ryan Kaldari <[hidden email]>
wrote:

> Nice work guys! One slight issue though. The user Selenium_user (as set in
> environment_variables) was just indefinitely blocked on en.wiki :(
>

Yes, and I think we should keep it that way. See my msg. to the mobile-l
and qa lists earlier today:
http://lists.wikimedia.org/pipermail/mobile-l/2014-June/007435.html

These tests are intended as acceptance tests for a new mediawiki install
and smoke tests for a test env like beta labs.  Let's not run them against
production.

-Chris



>
> Ryan Kaldari
>
>
> On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp <[hidden email]>
> wrote:
>
> > I just +2'ed a change to add a few basic selenium tests to core [1]. I
> > think it will benefit us all to have a set of automated tests to
> > quickly make sure mediawiki is working correctly. From a security
> > perspective, this also takes a step towards more efficient security
> > testing, which I'm also a fan of (if you've tried blindly scanning
> > mediawiki, you know what I'm talking about..).
> >
> > I think the QA group is working on vagrant-izing these, but if you
> > have ruby >1.9.3 and firefox, then setting up and running these tests
> > on your local dev system is 4-6 commands,
> >
> > $ cd tests/browser
> > $ gem update --system
> > $ gem install bundler
> > $ bundle install
> >
> > You can either set your environment variables yourself, or edit
> > environment_variables and run `source environment_variables` to set
> > them. Then it's just
> >
> > $ bundle exec cucumber features/
> >
> > to run the tests. They currently complete in 36 seconds on my laptop.
> >
> > I'd like to see more tests added and backported to REL1_23 to make
> > sure we have an ongoing suite to check releases against for next few
> > years that we support that LTS. If anyone is interested in both
> > mediawiki core and browser tests, I'm sure the QA team would like to
> > get you involved.
> >
> > Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> > and getting this done. I'll let them jump in with all the details I've
> > missed.
> >
> >
> > [1] - https://gerrit.wikimedia.org/r/#/c/133507/
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Dan Duvall
In reply to this post by Chris Steipp
Thanks for reviewing that, Chris!

Chris McMahon did most of the hard work there, porting everything from
qa/browsertests into core. I was glad to help on the Vagrant side of
things, though: I have a mediawiki-vagrant commit standing by that will do
all the heavy heavy installation lifting that you mentioned. :) I'll double
check that it's backwards compatible and send it up for review shortly.



On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp <[hidden email]> wrote:

> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly. From a security
> perspective, this also takes a step towards more efficient security
> testing, which I'm also a fan of (if you've tried blindly scanning
> mediawiki, you know what I'm talking about..).
>
> I think the QA group is working on vagrant-izing these, but if you
> have ruby >1.9.3 and firefox, then setting up and running these tests
> on your local dev system is 4-6 commands,
>
> $ cd tests/browser
> $ gem update --system
> $ gem install bundler
> $ bundle install
>
> You can either set your environment variables yourself, or edit
> environment_variables and run `source environment_variables` to set
> them. Then it's just
>
> $ bundle exec cucumber features/
>
> to run the tests. They currently complete in 36 seconds on my laptop.
>
> I'd like to see more tests added and backported to REL1_23 to make
> sure we have an ongoing suite to check releases against for next few
> years that we support that LTS. If anyone is interested in both
> mediawiki core and browser tests, I'm sure the QA team would like to
> get you involved.
>
> Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> and getting this done. I'll let them jump in with all the details I've
> missed.
>
>
> [1] - https://gerrit.wikimedia.org/r/#/c/133507/
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




--
Dan Duvall
Automation Engineer
Wikimedia Foundation <http://wikimediafoundation.org>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Risker
In reply to this post by Ryan Kaldari-2
On 24 June 2014 18:56, Ryan Kaldari <[hidden email]> wrote:

> Nice work guys! One slight issue though. The user Selenium_user (as set in
> environment_variables) was just indefinitely blocked on en.wiki :(
>
>

Sorry to be a bit OT, but if you guys are going to test, please don't do it
in article space on enwiki, or this is what is going to happen to the
accounts.  We've had to almost kick WMF staff off enwiki before because
they kept testing in live article space, please don't do that.

Just be glad that the blocking admin didn't make it a block with IP
autoblocked for 24 hours, or there would have been a bigger problem.

Testwiki is for test....and if you must test on enwiki, do it in userspace.

Risker/Anne
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Brian Wolff
On Jun 24, 2014 9:05 PM, "Risker" <[hidden email]> wrote:
>
> On 24 June 2014 18:56, Ryan Kaldari <[hidden email]> wrote:
>
> > Nice work guys! One slight issue though. The user Selenium_user (as set
in
> > environment_variables) was just indefinitely blocked on en.wiki :(
> >
> >
>
> Sorry to be a bit OT, but if you guys are going to test, please don't do
it
> in article space on enwiki, or this is what is going to happen to the
> accounts.  We've had to almost kick WMF staff off enwiki before because
> they kept testing in live article space, please don't do that.
>
> Just be glad that the blocking admin didn't make it a block with IP
> autoblocked for 24 hours, or there would have been a bigger problem.
>
> Testwiki is for test....and if you must test on enwiki, do it in
userspace.
>
> Risker/Anne
>

That sounds entirely reasonable to me.

--bawolff
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Dan Garry
In reply to this post by Risker
On 24 June 2014 17:05, Risker <[hidden email]> wrote:
>
> Sorry to be a bit OT, but if you guys are going to test, please don't do it
> in article space on enwiki, or this is what is going to happen to the
> accounts.  We've had to almost kick WMF staff off enwiki before because
> they kept testing in live article space, please don't do that.
>

Or you'll accidentally insert vandalism into articles when testing the
abuse filter. Because I've *totally never* done that by accident. :-)

If these browsers tests don't interact with the site that leaves anything
user-facing behind (e.g. making edits or take actions that create log
entries), there's no problem with running them on enwiki. Otherwise, they
should be using our test or beta sites.

Dan

--
Dan Garry
Associate Product Manager for Platform and Mobile Apps
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Chris Steipp
On Jun 24, 2014 6:13 PM, "Dan Garry" <[hidden email]> wrote:
>
> On 24 June 2014 17:05, Risker <[hidden email]> wrote:
> >
> > Sorry to be a bit OT, but if you guys are going to test, please don't
do it

> > in article space on enwiki, or this is what is going to happen to the
> > accounts.  We've had to almost kick WMF staff off enwiki before because
> > they kept testing in live article space, please don't do that.
> >
>
> Or you'll accidentally insert vandalism into articles when testing the
> abuse filter. Because I've *totally never* done that by accident. :-)
>
> If these browsers tests don't interact with the site that leaves anything
> user-facing behind (e.g. making edits or take actions that create log
> entries), there's no problem with running them on enwiki. Otherwise, they
> should be using our test or beta sites.

They do leave some artifacts (documented in the change, iirc). So yeah,
they should only be used against your dev environment or beta.

>
> Dan
>
> --
> Dan Garry
> Associate Product Manager for Platform and Mobile Apps
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Browser tests for core

Antoine Musso-3
In reply to this post by Chris Steipp
Le 25/06/2014 00:46, Chris Steipp a écrit :
> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly. From a security
> perspective, this also takes a step towards more efficient security
> testing, which I'm also a fan of (if you've tried blindly scanning
> mediawiki, you know what I'm talking about..).
<snip>

The next big step would be to trigger them whenever someone propose a
patchset in Gerrit or vote +2.  We have some scripts to setup a
MediaWiki locally and run the browser tests, but that proven to be a bit
unstable :-/

I am wondering how developer would feel with having browser tests
reported back to Gerrit.  Possibly has a different check so we dont have
to wait for them to complete before reporting all the other tests.



> I'd like to see more tests added and backported to REL1_23 to make
> sure we have an ongoing suite to check releases against for next few
> years that we support that LTS. If anyone is interested in both
> mediawiki core and browser tests, I'm sure the QA team would like to
> get you involved.

That is a nice idea.  We could then trigger them before releasing a new
tarball which will add some confidence.  There is a bit of script needed
there but that is definitely doable.

> Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> and getting this done. I'll let them jump in with all the details I've
> missed.

Oh I have done nothing beside requesting someone to write the
announcement :-]

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l