2016-06-01 Scrum of Scrums meeting notes

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

2016-06-01 Scrum of Scrums meeting notes

Grace Gellerman
https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-06-01


= 2016-06-01 =


== Product ==

=== Reading ===

==== Reading Infrastructure ====

* AuthManager coming soon - it's on the beta cluster now! Please indicate
if you hae a serious blocker now at
https://phabricator.wikimedia.org/T135504

==== Web ====
* Starting Hovercards A/B test during this sprint (with 1% in HU.wiki)
* Rolling lazy loaded images / references to medium sized wikis
* Blocked (Ops/Analytics) - https://phabricator.wikimedia.org/T127883

==== iOS ====

==== Android ====
* Production release 2.2.146 rollout out this Tuesday.
https://lists.wikimedia.org/pipermail/mobile-l/2016-May/010243.html
  (New features include: reading lists, edit here, Show redirect source in
search results, updates for AuthManager,
  No longer download and widen high-quality images when on a metered
connection.)
* Working on new feed UI

==== Mobile Content Service ====
* Adding first couple of new feed endpoints this week (enwiki featured
article, most read articles, aggregated endpoint for previous and future
feed microservices)

=== Community Tech ===
* We're going to be testing the PageAssessments extension on Labs this week
(https://www.mediawiki.org/wiki/Extension:PageAssessments)
* Continuing work on new CopyPatrol Tool (
http://tools.wmflabs.org/plagiabot)
* More investigation on Category collation
* No blockers

=== Editing ===

==== Language ====
* Blocked: Parallel Corpora dumps:
https://phabricator.wikimedia.org/T127793 (Tech
Ops)

==== Multimedia ====
* Blocked: New repository request for FileAnnotations so we can experiment
and do design/code review on Mark's crappy prototype
* Updates
** ImageTweaks waiting on Thumbor and a few other things, no news there
AFAIK
** Gallery slideshow core work coming along, no huge updates from prtksxna
** UW fatals rate waaaay down since last year, thanks Sentry people for
helping us track it

==== Parsing ====
* Tidy replacement: With latest HTML5Depurate and visual diff test run,
~93.4% of test pages render with pixel perfect accuracy. Working through
remaining diff scenarios (a lot of the diffs are harmless minor pixel
shifts) to identify real issues and having wikitext be fixed or fix core
parser as required.
** Tim working on a HTML5 parser to fix some core parser issues around
doBlockLevels.
* Will attempt switching Parsoid to service-runner sometime next week (Arlo
& Services)
* Arlo close to getting html2html endpoint done (last remaining blocker to
enable Parsoid new HTML version with separate data-mw)
** VE & CX: Please start planning how to handle this new version.
* Kunal working on using the new Linker code for files/images.

==== Collaboration ====
* Blocked - None
* Blocking - No change
* Updates
** Working on Echo special page, for both JS and no-JS users
** Schema changes we requested are done, we now need to backfill and use
the new schema

== Technology ==

=== Services ===
* RESTBase
** enforcing rate limits as of today
*** pageview: 10 req/s
*** transforms: 5 req/s
* Change Propagation
** now handles summary and MCS updates as well as purges
** partially moved RestbaseUpdateJobs from job runners to it
*** still missing transclusions
* MathML enabled by default across all projects
** proper caching and purging still a TODO
*** RESTBase side for purging proposed in
https://github.com/wikimedia/restbase/pull/628
*** need Ops help for the Varnish side of things -
https://phabricator.wikimedia.org/T136205#2343850

=== Technical Operations ===
* '''Blocking''': none
* '''Blocked''': none
* Updates:
** libicu update successful
** Started preliminary work on migrating appservers to jessie
** Last day of Riccardo
** Getting graphite into better shape

=== Analytics ===
* Loaded 3 months of pageview data into Druid and querying it is very fast
* Cleaning up limn-flow-data, limn-edit-data, and limn-languge-data a
little bit, deploying today
* Working on processing data from the mediawiki databases and turning it
into an analytics-friendly schema (one example is slowly changing
dimensions such as page_title recorded as (page_id, page_title, valid_from,
valid_to))
* pageview requests that miss the cache are rate-limited to 10 req/s.  More
req/s than that will throw 429

=== Security ===
* Two-factor usability surveying prep continues
* Reviews: Ex:DoubleWiki, Ex:TemplateStyles, Use of wheels for deployment
by Analytics

=== Discovery ===
* '''Blocking''': none
* '''Blocked''': none
* Tomasz last day at WMF was on Friday, Katie Horn leading Discovery now
* ElasticSearch 2.3 upgrade in progress:
https://phabricator.wikimedia.org/T133124
* TextCat A/B test for english wiki started
* Updated Portal language stats.

=== Fundraising Tech ===
* Fixed CentralNotice tests, now enforcing ES3
* More work to get off ActiveMQ, remove SPOF
* Testing new processor in Israel, Japan and Ukraine
** We quit messing with language fallback and just got the last few missing
messages translated to Ukrainian
* Ready to test new PayPal integration method
* Enhancing fraud & dos mitigation measures

=== Release Engineering ===
* '''Blocking''': None
* '''Blocked''': Nada
* '''Update:'''
** Auth Manager rolling out *soon*
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: 2016-06-01 Scrum of Scrums meeting notes

Pine W
Hi Grace,

I'd like to request a clarification about RESTBase.

These notes say:
"* RESTBase
** enforcing rate limits as of today
*** pageview: 10 req/s
*** transforms: 5 req/s"

Can you expand on what kinds of requests these rate limits are limiting?

Thanks,
Pine
On Jun 1, 2016 11:06, "Grace Gellerman" <[hidden email]> wrote:

> https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-06-01
>
>
> = 2016-06-01 =
>
>
> == Product ==
>
> === Reading ===
>
> ==== Reading Infrastructure ====
>
> * AuthManager coming soon - it's on the beta cluster now! Please indicate
> if you hae a serious blocker now at
> https://phabricator.wikimedia.org/T135504
>
> ==== Web ====
> * Starting Hovercards A/B test during this sprint (with 1% in HU.wiki)
> * Rolling lazy loaded images / references to medium sized wikis
> * Blocked (Ops/Analytics) - https://phabricator.wikimedia.org/T127883
>
> ==== iOS ====
>
> ==== Android ====
> * Production release 2.2.146 rollout out this Tuesday.
> https://lists.wikimedia.org/pipermail/mobile-l/2016-May/010243.html
>   (New features include: reading lists, edit here, Show redirect source in
> search results, updates for AuthManager,
>   No longer download and widen high-quality images when on a metered
> connection.)
> * Working on new feed UI
>
> ==== Mobile Content Service ====
> * Adding first couple of new feed endpoints this week (enwiki featured
> article, most read articles, aggregated endpoint for previous and future
> feed microservices)
>
> === Community Tech ===
> * We're going to be testing the PageAssessments extension on Labs this week
> (https://www.mediawiki.org/wiki/Extension:PageAssessments)
> * Continuing work on new CopyPatrol Tool (
> http://tools.wmflabs.org/plagiabot)
> * More investigation on Category collation
> * No blockers
>
> === Editing ===
>
> ==== Language ====
> * Blocked: Parallel Corpora dumps:
> https://phabricator.wikimedia.org/T127793 (Tech
> Ops)
>
> ==== Multimedia ====
> * Blocked: New repository request for FileAnnotations so we can experiment
> and do design/code review on Mark's crappy prototype
> * Updates
> ** ImageTweaks waiting on Thumbor and a few other things, no news there
> AFAIK
> ** Gallery slideshow core work coming along, no huge updates from prtksxna
> ** UW fatals rate waaaay down since last year, thanks Sentry people for
> helping us track it
>
> ==== Parsing ====
> * Tidy replacement: With latest HTML5Depurate and visual diff test run,
> ~93.4% of test pages render with pixel perfect accuracy. Working through
> remaining diff scenarios (a lot of the diffs are harmless minor pixel
> shifts) to identify real issues and having wikitext be fixed or fix core
> parser as required.
> ** Tim working on a HTML5 parser to fix some core parser issues around
> doBlockLevels.
> * Will attempt switching Parsoid to service-runner sometime next week (Arlo
> & Services)
> * Arlo close to getting html2html endpoint done (last remaining blocker to
> enable Parsoid new HTML version with separate data-mw)
> ** VE & CX: Please start planning how to handle this new version.
> * Kunal working on using the new Linker code for files/images.
>
> ==== Collaboration ====
> * Blocked - None
> * Blocking - No change
> * Updates
> ** Working on Echo special page, for both JS and no-JS users
> ** Schema changes we requested are done, we now need to backfill and use
> the new schema
>
> == Technology ==
>
> === Services ===
> * RESTBase
> ** enforcing rate limits as of today
> *** pageview: 10 req/s
> *** transforms: 5 req/s
> * Change Propagation
> ** now handles summary and MCS updates as well as purges
> ** partially moved RestbaseUpdateJobs from job runners to it
> *** still missing transclusions
> * MathML enabled by default across all projects
> ** proper caching and purging still a TODO
> *** RESTBase side for purging proposed in
> https://github.com/wikimedia/restbase/pull/628
> *** need Ops help for the Varnish side of things -
> https://phabricator.wikimedia.org/T136205#2343850
>
> === Technical Operations ===
> * '''Blocking''': none
> * '''Blocked''': none
> * Updates:
> ** libicu update successful
> ** Started preliminary work on migrating appservers to jessie
> ** Last day of Riccardo
> ** Getting graphite into better shape
>
> === Analytics ===
> * Loaded 3 months of pageview data into Druid and querying it is very fast
> * Cleaning up limn-flow-data, limn-edit-data, and limn-languge-data a
> little bit, deploying today
> * Working on processing data from the mediawiki databases and turning it
> into an analytics-friendly schema (one example is slowly changing
> dimensions such as page_title recorded as (page_id, page_title, valid_from,
> valid_to))
> * pageview requests that miss the cache are rate-limited to 10 req/s.  More
> req/s than that will throw 429
>
> === Security ===
> * Two-factor usability surveying prep continues
> * Reviews: Ex:DoubleWiki, Ex:TemplateStyles, Use of wheels for deployment
> by Analytics
>
> === Discovery ===
> * '''Blocking''': none
> * '''Blocked''': none
> * Tomasz last day at WMF was on Friday, Katie Horn leading Discovery now
> * ElasticSearch 2.3 upgrade in progress:
> https://phabricator.wikimedia.org/T133124
> * TextCat A/B test for english wiki started
> * Updated Portal language stats.
>
> === Fundraising Tech ===
> * Fixed CentralNotice tests, now enforcing ES3
> * More work to get off ActiveMQ, remove SPOF
> * Testing new processor in Israel, Japan and Ukraine
> ** We quit messing with language fallback and just got the last few missing
> messages translated to Ukrainian
> * Ready to test new PayPal integration method
> * Enhancing fraud & dos mitigation measures
>
> === Release Engineering ===
> * '''Blocking''': None
> * '''Blocked''': Nada
> * '''Update:'''
> ** Auth Manager rolling out *soon*
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: 2016-06-01 Scrum of Scrums meeting notes

James Forrester-4
On 3 June 2016 at 00:14, Pine W <[hidden email]> wrote:

> I'd like to request a clarification about RESTBase.
>
> These notes say:
> "* RESTBase
> ** enforcing rate limits as of today
> *** pageview: 10 req/s
> *** transforms: 5 req/s"
>
> Can you expand on what kinds of requests these rate limits are limiting?
>

​The first one is to the page content requests​, documented at
https://en.wikipedia.org/api/rest_v1/?doc#/Page_content.

The second one, similarly, is to content transformation requests, shown at
https://en.wikipedia.org/api/rest_v1/?doc#/Transforms.

As always, our first mission is to keep the sites up and running well; as
with other API access points, this means dealing with abusive requests
where we see them, and (as in this case) limiting access to a reasonable
level to avoid overloading the system. Unless you're running a tool against
this API at a very high speed, ignoring the Terms of Use, it won't affect
you.

​J.
--
James D. Forrester
Lead Product Manager, Editing
Wikimedia Foundation, Inc.

[hidden email] | @jdforrester
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: 2016-06-01 Scrum of Scrums meeting notes

Gabriel Wicke-3
On Fri, Jun 3, 2016 at 10:21 AM, James Forrester
<[hidden email]> wrote:

> On 3 June 2016 at 00:14, Pine W <[hidden email]> wrote:
>
>> I'd like to request a clarification about RESTBase.
>>
>> These notes say:
>> "* RESTBase
>> ** enforcing rate limits as of today
>> *** pageview: 10 req/s
>> *** transforms: 5 req/s"
>>
>> Can you expand on what kinds of requests these rate limits are limiting?

Small correction: The "pageview" reference here is to the page view
API end points:

- https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data/get_metrics_pageviews_per_article_project_access_agent_article_granularity_start_end
- https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data/get_metrics_pageviews_aggregate_project_access_agent_granularity_start_end
- https://wikimedia.org/api/rest_v1/?doc#!/Pageviews_data/get_metrics_pageviews_top_project_access_year_month_day

See https://phabricator.wikimedia.org/T135240 for background on these
limits on the pageview API. tl;dr: The backend for pageview data has
limited traffic capacity right now, but the analytics team is working
on a hardware upgrade. In the meantime, we need to make sure that the
limited capacity is used fairly, by enforcing conservative per-client
limits.

> Unless you're running a tool against
> this API at a very high speed, ignoring the Terms of Use, it won't affect
> you.

Indeed. For the (relatively expensive) transform end points, we picked
the limits so that none of the current users are actually affected by
them.

Gabriel

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l