Unit tests scream for attention

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Unit tests scream for attention

Niklas Laxström
Now that tests need +2 to be run, at least temporarily, I'm going to
point out that I've not been able to run tests on my development
environment in ages. I mentioned broken unit tests in Oct 4 on this
list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390

There are multiple fatal bugs (not to mention the numerous test
failures), that halt test runs without any info expect the error. Some
bugs I've reported:

* https://bugzilla.wikimedia.org/41491
* https://bugzilla.wikimedia.org/42145 (skip the first few comments)

Today I tried again and there is new one:

Catchable fatal error: Argument 2 passed to
OutputPage::addWikiTextTitle() must be an instance of Title, null
given, called in /www/dev.translatewiki.net/w/includes/OutputPage.php
on line 1426 and defined in
/www/dev.translatewiki.net/w/includes/OutputPage.php on line 1472

This might be just a variant of 42145, but I can't tell for sure. I
could add exception there, but the other fatal errors make phpunit not
to display backtraces. I haven't yet had time to try to find out which
test it is.

This situation is starting to feel like a bad horror movie, so I ask
everyone to give some tender, love and care to our unit tests so that
I don't have to come up with even worse analogies.

  -Niklas

--
Niklas Laxström

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Antoine Musso-3
Le 07/12/12 19:13, Niklas Laxström a écrit :
> Now that tests need +2 to be run, at least temporarily, I'm going to
> point out that I've not been able to run tests on my development
> environment in ages. I mentioned broken unit tests in Oct 4 on this
> list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390
>
> There are multiple fatal bugs (not to mention the numerous test
> failures), that halt test runs without any info expect the error.
<snip>

Hello Niklas,

I will be more than happy to help you track down the root cause of such
failures.  It seems your setup has lot of extensions loaded in and most
probably a non default LocalSettings.php.

To track the issue down you could start by disabling all extensions and
see how it goes, then enable extension one by one and rerun tests in
between.

Same goes for settings, try out with a fresh install of just MediaWiki
core then add the settings one by one and see what happens.

cheers,

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Brad Jorsch (Anomie)
In reply to this post by Niklas Laxström
On Fri, Dec 7, 2012 at 1:13 PM, Niklas Laxström
<[hidden email]> wrote:
> There are multiple fatal bugs (not to mention the numerous test
> failures), that halt test runs without any info expect the error. Some
> bugs I've reported:
>
> * https://bugzilla.wikimedia.org/41491

While that unit test mentioned there does seem screwed up, why is your
PHPUnit installation not respecting convertErrorsToExceptions="true"
and related settings in MediaWiki's provided suite.xml?

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Niklas Laxström
On 7 December 2012 22:51, Brad Jorsch <[hidden email]> wrote:
> While that unit test mentioned there does seem screwed up, why is your
> PHPUnit installation not respecting convertErrorsToExceptions="true"
> and related settings in MediaWiki's provided suite.xml?

I honestly don't know. I barely got working PHPUnit installed in the
first place (3.7.8 - see my other previous phpunit thread). I also
don't know why it occasionally segfaults.

  -Niklas


--
Niklas Laxström

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Platonides
In reply to this post by Niklas Laxström
They mostly run for me.

Those failing are:
- PreferencesTest and TitleMethodsTest requiring a DB but not in
Database group.
- SanitizerTest::testRemovehtmltagsOnHtml5Tags skipping the closing tag
an <data>, <mark> and <time> (???)
- IPTCTest::testIPTCParseForcedUTFButInvalid with php 5.4. Reported in
June as bug 37665

Plus an odd error by DatabaseTest::testStoredFunctions



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Niklas Laxström
On 8 December 2012 21:22, Platonides <[hidden email]> wrote:
> They mostly run for me.

Yep, mostly:
Tests: 4653, Assertions: 556494, Failures: 119, Errors: 17,
Incomplete: 3, Skipped: 15.

I've not even looked at most of these failures because of the fatal
errors. Thanks to Jeroen and Brad there is now one fatal error less
(and at least one remains), though I did spend two hours debugging
that issue.
  -Niklas

--
Niklas Laxström

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Sumana Harihareswara-2
In reply to this post by Niklas Laxström
On 12/07/2012 01:13 PM, Niklas Laxström wrote:
> Now that tests need +2 to be run, at least temporarily, I'm going to
> point out that I've not been able to run tests on my development
> environment in ages. I mentioned broken unit tests in Oct 4 on this
> list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390

Niklas, are you still having these problems, or are they mostly resolved?

The ideal is for all the regular developers to automatically run the
test suite locally before submitting changesets, so if anyone's had
problems that stop them from doing that, we ought to learn why, and fix
those obstacles.

--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

bawolff
On Thu, Dec 27, 2012 at 11:48 PM, Sumana Harihareswara
<[hidden email]> wrote:

> On 12/07/2012 01:13 PM, Niklas Laxström wrote:
>> Now that tests need +2 to be run, at least temporarily, I'm going to
>> point out that I've not been able to run tests on my development
>> environment in ages. I mentioned broken unit tests in Oct 4 on this
>> list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390
>
> Niklas, are you still having these problems, or are they mostly resolved?
>
> The ideal is for all the regular developers to automatically run the
> test suite locally before submitting changesets, so if anyone's had
> problems that stop them from doing that, we ought to learn why, and fix
> those obstacles.


I used to run unit tests at regular intervals. I stopped because it is
such a pain to install a version of phpunit that actually works (Also
its less critical to do it yourself now that jenkins does it for you).

When I used to run unit tests, there were quite regularly issues where
the unit tests assumed you had the default configuration, where they
really should not assume such a thing. (That was of course a while
ago, so things may have changed).

--bawolff

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Bryan Tong Minh
On Sat, Dec 29, 2012 at 8:57 AM, bawolff <[hidden email]> wrote:

>
> When I used to run unit tests, there were quite regularly issues where
> the unit tests assumed you had the default configuration, where they
> really should not assume such a thing. (That was of course a while
> ago, so things may have changed).
>

This is an annoyance to me as well. So, I went triaging, and finally found
the issues that failed the unit tests for me.  I have committed fixes for
them to gerrit:
https://gerrit.wikimedia.org/r/#/c/41362/
https://gerrit.wikimedia.org/r/#/c/41360/

Bryan
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Unit tests scream for attention

Chris McMahon
On Sat, Dec 29, 2012 at 1:47 PM, Bryan Tong Minh
<[hidden email]>wrote:

>
> This is an annoyance to me as well. So, I went triaging, and finally found
> the issues that failed the unit tests for me.  I have committed fixes for
> them to gerrit:
> https://gerrit.wikimedia.org/r/#/c/41362/
> https://gerrit.wikimedia.org/r/#/c/41360/
>

This sort of thing has come up before.  Michael Feathers, author of Working
Effectively with Legacy Code (
http://www.artima.com/weblogs/viewpost.jsp?thread=126923), etc. published a
guideline for unit tests in 2005:
http://www.artima.com/weblogs/viewpost.jsp?thread=126923:

A test is not a unit test if:

It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config
files) to run it.

I am not a great writer of unit tests, especially in PHP, but it is my
impression that we have quite a few existing tests that don't follow this
guide.  In Bryan's example above, having a unit test depend on the
existence of gzip would be a code smell.

I don't think it's an immediate priority, but over time it would probably
make sense to refactor the tests that don't follow this guide to use mocks
and stubs and other accepted practices to avoid requiring a particular
environment to run unit tests.
-Chris
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l