2017-11-01 Scrum of Scrums meeting notes

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

2017-11-01 Scrum of Scrums meeting notes

Grace Gellerman
https://www.mediawiki.org/wiki/Scrum_of_scrums/2017-11-01


= 2017-11-01=
Callouts:
* If you notice anything weird with VE, let Services know, they are messing
with Parsoid storage

== Audiences ==

=== Readers ===

==== Multimedia ====
* Blocking: None
* Blocked by: None
* Updates: MediaInfo/Wikibase on Commons work is continuing,
https://phabricator.wikimedia.org/T177022
* UploadWizard changes for 3D legal considerations,
https://phabricator.wikimedia.org/T178513
* and discussions about MP3.
* Heads up: Mark on vacation until Thursday next week - will respond but
may be AFK

==== iOS native app ====
* Blocked by:
* Blocking:
* Updates:
** 5.7.0 released ( https://phabricator.wikimedia.org/project/view/2899/ )
- Visual updates to onboarding, iOS 11 support, iPhone X support
** Starting work on 5.7.1 -
https://phabricator.wikimedia.org/project/view/3047/ (bug fixes)  and 5.8 -
https://phabricator.wikimedia.org/project/view/2913/ (faster article loads,
synced reading lists)

==== Discovery ====
* Blocked by:
* Blocking:
* Updates:
** Lots up SVG related patch to wikipedia portal

https://gerrit.wikimedia.org/r/#/q/topic:new-svg-sprites+(status:open+OR+status:merged)

===== Maps =====
* Blocked by: N/A
* Blocking: N/A
* Updates:
** People are back from SOTM US
** Lots of relevant conversations relevant to our maps at SOTM US

==== Reading Infrastructure ====
* Blocked by:
* Blocking:
* Updates:
** MCS getting ready for Parsoid section tags (in mobile-sections and
Wiktionary definitions). Discussions about wrapping the sections content in
a <div> on https://phabricator.wikimedia.org/T176959.
** MCS adding more title related properties to summary own implementation
and mobile-sections lead portion.
** Continuing work on making Reading Lists production-ready

==== Services ====
* Blockers: none
* Updates:
** Testing switching Parsoid to new storage in labs and production
** Summary will be served from new storage only starting tomorrow

=== Contributors ===

==== Global Collaboration ====
* Blocked by: none
* Blocking: none
* Updates:
** Team offsite in Bangalore last week, not everyone's back yet
** Actually deployed Compact Language Links to dewiki last week
(2017-10-25); was canceled the week before

==== Parsing ====
* Updates:
** Updating section wrapping code to not add the div wrapper around section
content as per discussion in https://phabricator.wikimedia.org/T176959
** Barring no other unforeseen issues, might be able to deploy this patch
next week along with a couple other changes: (a) html5 section ids to match
core changes (b) wrapping inline images using figure-inline tag; We'll bump
our HTML version number at that time

==== Community Tech ====
* New wishlist survey starts next week
* Global Preferences RFC will be discussed today
* Deployed Unicode sections to Russian projects, addressing issues

==== Fundraising Tech ====
* Tweaks to CiviCRM reports, searches, and contact views
* Deployed various minor CentralNotice code changes:
https://gerrit.wikimedia.org/r/387627
* Backup card processor: still fixing Safari cookie issue, also adding
retry on failure to certain API calls
* Refactoring initial prometheus stats collector, adding more measurements

=== Search Platform ===
* Blocked by: none
* Blocking: none
* Updates:
* Finished evaluating MLR training resource usage, updated docs:
https://github.com/wikimedia/search-MjoLniR/blob/master/docs/running-in-analytics.rst
* Fixed search matches within section titles (
https://phabricator.wikimedia.org/T178522)
* Fixing issues between completion suggester and multi-ns searches on some
wikis (https://phabricator.wikimedia.org/T178474)
* Enabled ElasticSearch prefix search on Wikidata, dealing with some
special case regressions
* Looking at top abandoned queries (
https://phabricator.wikimedia.org/T176997)
* Working on porting Selenium tests from Ruby to JS
* Working on upgrade to Elastic 5.5
* Added indexing Wikidata statements for P31 and P279
* Working on indexing Wikidata descriptions and adding them to fulltext
search

=== Scoring Platform ===
* Blocked by:
* Blocking:
* Updates:
** Shiny new FAQ!
*** https://www.mediawiki.org/wiki/ORES/FAQ
** Converted our tests from nosetests to pytest (Anyone else making the
transition?)
*** nosetests is under-maintained
*** python community is moving towards pytest
*** converting our repos was pretty straightforward


=== Release Engineering ===
* Blocking: None
* Blocked: None
* Updates:
** We had to pause the MediaWiki train last week due to a hard to diagnose
issue in production: *https://phabricator.wikimedia.org/T179156*
<https://phabricator.wikimedia.org/T179156>
** T171852 Tech talk: Selenium tests in Node.js
https://phabricator.wikimedia.org/phame/post/view/78/tech_talk_selenium_tests_in_node.js/
** T173488 Selenium Ruby framework deprecated
https://phabricator.wikimedia.org/phame/post/view/79/selenium_ruby_framework_deprecation_october/
** CI: Antoine was sick last week. But, he setup a simple package manager
caching system for the docker based CI (a port of “castor” in our nodepool
based CI). He will migrate some of the CI config related tox jobs to it.

=== Security ===
* Blocking: all who are waiting for security reviews
* Blocked: none
* Updates:
** Reviews
*** BeautifulSoup
*** mediawiki-services-chromium-render
*** Marvin
*** Ex:FIleImporter
*** Ex:WikipediaExtracts
*** Onboarding Erika Bjune to team

=== Analytics ===
* new host replaces db1047, EventLogging breathes a sigh of relief
* continuing work on jumbo kafka cluster, EventLogging refine, mediawiki
history final data vetting, Wikistats 2 alpha release
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Migrating away from nose (was 2017-11-01 Scrum of Scrums meeting notes)

Kunal Mehta
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

On 11/01/2017 10:53 AM, Grace Gellerman wrote:
> === Scoring Platform === ** Converted our tests from nosetests to
> pytest (Anyone else making the transition?) *** nosetests is
> under-maintained *** python community is moving towards pytest ***
> converting our repos was pretty straightforward

I didn't realize nose was in such a state :( They put a notice[1] on
their website, but I always skipped it.

pytest[2] claims that it's compatible with nose out of the box - was
that the experience you had as well?

Off the top of my head, Pywikibot, CI, and other miscellaneous
projects all use nose and should probably be migrated over to pytest.

[1] https://nose.readthedocs.io/en/latest/#note-to-users
[2] https://docs.pytest.org/en/latest/

- -- Legoktm
-----BEGIN PGP SIGNATURE-----

iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAln6D0gXHGxlZ29rdG1A
bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KKbMw/+Nh/tGRagrvZgJqHBZakvvAYE
IHCUFPTUhfQ4cQ++oUqDpblAxR/FNO6BN2UQIrIh9gUylHCf/hwZ3BToC5GGPFY5
D9Sgi/ce9v5dp3IIvNZq/NZgb0DkityVkxWg8tiZJPwhJ6hTKw9LjaPl++ee9Po/
940WYy7UQ7L/xwgy7rNJrViW0j4VD4mM7+PVvxPuv6rovOpa/H0VMbSlEuoXQYFG
P2DPJE1f8B9tn/xun6O6tveWdD7cdna2DVCXhFF+Tldnpa+XtOerE3P+YBA8Gp/+
sRJZuDjbCljM0GCzYYVR00klp8xDFQJXn+ELWaE1E3pAitZQdjb1A+xPmxagk1Up
ysqSc9nndVm6c59lkzdMcqz1myWyzLVho/RBmRVAAm37Wv7F7tvxHufuwWe8QO/I
sUJsRXkH7/TdsSzs8TU9YTZECE5QntTOM7gGgP4jPQgiwZrHob9si1jDUaQMuFoC
i8U9dylhrh280VAJj9LRaoRaSb48Ln6sZrCMdpCW8gt4UiRsUcSYCXbD45AgUS9Z
3oPLMEpt+ESqDxJrBl9rgxyFZF19KhpeE3FPCvlAMZQShLZCAlRdDozczV4nFSHC
zhRmuRITmnDomf8prk7NwFSM0q601kTOgVqVZiBn9AY3Axvsa1TS1HM582y4rJ8U
O+eQbaSCyiI6HTE0lcc=
=4lTI
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Migrating away from nose (was 2017-11-01 Scrum of Scrums meeting notes)

Greg Grossmeier-2
(adding in qa@ list)

<quote name="Kunal Mehta" date="2017-11-01" time="11:15:42 -0700">

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Hi,
>
> On 11/01/2017 10:53 AM, Grace Gellerman wrote:
> > === Scoring Platform === ** Converted our tests from nosetests to
> > pytest (Anyone else making the transition?) *** nosetests is
> > under-maintained *** python community is moving towards pytest ***
> > converting our repos was pretty straightforward
>
> I didn't realize nose was in such a state :( They put a notice[1] on
> their website, but I always skipped it.
>
> pytest[2] claims that it's compatible with nose out of the box - was
> that the experience you had as well?
>
> Off the top of my head, Pywikibot, CI, and other miscellaneous
> projects all use nose and should probably be migrated over to pytest.
>
> [1] https://nose.readthedocs.io/en/latest/#note-to-users
> [2] https://docs.pytest.org/en/latest/
>
> - -- Legoktm
> -----BEGIN PGP SIGNATURE-----
>
> iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAln6D0gXHGxlZ29rdG1A
> bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KKbMw/+Nh/tGRagrvZgJqHBZakvvAYE
> IHCUFPTUhfQ4cQ++oUqDpblAxR/FNO6BN2UQIrIh9gUylHCf/hwZ3BToC5GGPFY5
> D9Sgi/ce9v5dp3IIvNZq/NZgb0DkityVkxWg8tiZJPwhJ6hTKw9LjaPl++ee9Po/
> 940WYy7UQ7L/xwgy7rNJrViW0j4VD4mM7+PVvxPuv6rovOpa/H0VMbSlEuoXQYFG
> P2DPJE1f8B9tn/xun6O6tveWdD7cdna2DVCXhFF+Tldnpa+XtOerE3P+YBA8Gp/+
> sRJZuDjbCljM0GCzYYVR00klp8xDFQJXn+ELWaE1E3pAitZQdjb1A+xPmxagk1Up
> ysqSc9nndVm6c59lkzdMcqz1myWyzLVho/RBmRVAAm37Wv7F7tvxHufuwWe8QO/I
> sUJsRXkH7/TdsSzs8TU9YTZECE5QntTOM7gGgP4jPQgiwZrHob9si1jDUaQMuFoC
> i8U9dylhrh280VAJj9LRaoRaSb48Ln6sZrCMdpCW8gt4UiRsUcSYCXbD45AgUS9Z
> 3oPLMEpt+ESqDxJrBl9rgxyFZF19KhpeE3FPCvlAMZQShLZCAlRdDozczV4nFSHC
> zhRmuRITmnDomf8prk7NwFSM0q601kTOgVqVZiBn9AY3Axvsa1TS1HM582y4rJ8U
> O+eQbaSCyiI6HTE0lcc=
> =4lTI
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| Release Team Manager            A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Migrating away from nose (was 2017-11-01 Scrum of Scrums meeting notes)

Riccardo Coccioli
I've migrated Cumin from 'nose' to 'pytest' a while ago, here's a
summary of my experience.

First of all we need to distinguish the two main parts of Python
tests: writing the tests and collecting and running them.

In the classical setup, 'unittest' would be used for writing the tests
and 'nose' for collecting and running them, often wrapped in a call to
'coverage'.
To use 'pytest' just as a test collector and runner, basically means
replacing 'nose' with 'pytest' and 'coverage' with the 'pytest-cov'
module and adding 'pytest-runner' to the 'setup_requires' in setup.py.
There are some caveats though, depending how many nose-specific
features are used and if any unsupported idiom is used, see also [1]
and [2].

Given the status of 'nose', I've looked at 'nose2', but it didn't seem
mature enough and actively developed. When I looked at it few months
ago that was no commit in the last 6-months and the latest release is
from mid-2016. I've seen that there has been some recent activity
though.
I then chose 'pytest' as I got the impression that the Python
community is slowly moving towards it. In the process I've also got
some quick and friendly help on their issue tracker.

I've done this migration in Cumin primarily because I wanted to
overcome some limitations in the previous setup, namely:
- more compact output of passing tests, listing dots per file and not
naming them all
- more verbose output on failing tests with the stacktrace, the code
involved and an easy way to collect separately stderr and stdout
- coverage plugin works with multiprocess testing (very useful for
integration tests) while nose+coverage did not work (known limitation)

Once this step is done, the other, optional but somehow natural step,
is to use 'pytest' also for writing the tests, replacing 'unittest'.
This could be partly automated by a tool like [3], although in my
experience it doesn't cover all the cases and needs some manual
cleanup, but surely helps.
This allows to take full advantage of all the pytest features, as some
are not compatible with unittest-based tests.

I've done this migration for the integration tests of Cumin to take
advantage of 'pytest' parametrization feature [4] that was very useful
in my case for integration tests that were automatically generated.

Once that was done, I've decided to migrate everything to 'pytest' because:
- coherence, to not have integration tests written with 'pytest' and
unit test written with 'unittest'
- to be able in the future to use advanced features of pytest in the
unittests, because they will most likely not work in unittest.TestCase
subclasses (and indeed it happened)
- style-wise 'unittest' doesn't follow PEP8

The main style-related drawback of using 'pytest' for unit tests is
that all the 'self.assertSomeThing(foo)' must be replaced by standard
asserts like 'assert foo == something', and the assert lines don't go
nicely multiline because is not possible to use parentheses. But this
can be easily avoided just assigning the expected and test result
values to variables just before doing the assert.

If you have specific questions while migrating your project, feel free
to contact me [5] or add me as code reviewer on Gerrit (user: Volans).

Riccardo

[1] https://docs.pytest.org/en/latest/nose.html
[2] https://docs.pytest.org/en/latest/unittest.html
[3] https://github.com/pytest-dev/unittest2pytest
[4] https://docs.pytest.org/en/latest/parametrize.html
[5] https://meta.wikimedia.org/wiki/User:RCoccioli_(WMF)


On Wed, Nov 1, 2017 at 7:34 PM, Greg Grossmeier <[hidden email]> wrote:

>
> (adding in qa@ list)
>
> <quote name="Kunal Mehta" date="2017-11-01" time="11:15:42 -0700">
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > Hi,
> >
> > On 11/01/2017 10:53 AM, Grace Gellerman wrote:
> > > === Scoring Platform === ** Converted our tests from nosetests to
> > > pytest (Anyone else making the transition?) *** nosetests is
> > > under-maintained *** python community is moving towards pytest ***
> > > converting our repos was pretty straightforward
> >
> > I didn't realize nose was in such a state :( They put a notice[1] on
> > their website, but I always skipped it.
> >
> > pytest[2] claims that it's compatible with nose out of the box - was
> > that the experience you had as well?
> >
> > Off the top of my head, Pywikibot, CI, and other miscellaneous
> > projects all use nose and should probably be migrated over to pytest.
> >
> > [1] https://nose.readthedocs.io/en/latest/#note-to-users
> > [2] https://docs.pytest.org/en/latest/
> >
> > - -- Legoktm
> > -----BEGIN PGP SIGNATURE-----
> >
> > iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAln6D0gXHGxlZ29rdG1A
> > bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KKbMw/+Nh/tGRagrvZgJqHBZakvvAYE
> > IHCUFPTUhfQ4cQ++oUqDpblAxR/FNO6BN2UQIrIh9gUylHCf/hwZ3BToC5GGPFY5
> > D9Sgi/ce9v5dp3IIvNZq/NZgb0DkityVkxWg8tiZJPwhJ6hTKw9LjaPl++ee9Po/
> > 940WYy7UQ7L/xwgy7rNJrViW0j4VD4mM7+PVvxPuv6rovOpa/H0VMbSlEuoXQYFG
> > P2DPJE1f8B9tn/xun6O6tveWdD7cdna2DVCXhFF+Tldnpa+XtOerE3P+YBA8Gp/+
> > sRJZuDjbCljM0GCzYYVR00klp8xDFQJXn+ELWaE1E3pAitZQdjb1A+xPmxagk1Up
> > ysqSc9nndVm6c59lkzdMcqz1myWyzLVho/RBmRVAAm37Wv7F7tvxHufuwWe8QO/I
> > sUJsRXkH7/TdsSzs8TU9YTZECE5QntTOM7gGgP4jPQgiwZrHob9si1jDUaQMuFoC
> > i8U9dylhrh280VAJj9LRaoRaSb48Ln6sZrCMdpCW8gt4UiRsUcSYCXbD45AgUS9Z
> > 3oPLMEpt+ESqDxJrBl9rgxyFZF19KhpeE3FPCvlAMZQShLZCAlRdDozczV4nFSHC
> > zhRmuRITmnDomf8prk7NwFSM0q601kTOgVqVZiBn9AY3Axvsa1TS1HM582y4rJ8U
> > O+eQbaSCyiI6HTE0lcc=
> > =4lTI
> > -----END PGP SIGNATURE-----
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | Release Team Manager            A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: 2017-11-01 Scrum of Scrums meeting notes

Sam Smith
In reply to this post by Grace Gellerman
On Wed, Nov 1, 2017 at 5:53 PM, Grace Gellerman <[hidden email]>
wrote:

> === Security ===
> * Updates:
> ** Reviews
> *** Marvin
>

A brief correction: I should've stalled this review request some time ago.
I've now done as much. My apologies for any confusion and/or delay that
this caused.

Thanks,

-Sam
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l