Production Excellence #22: June 2020

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Production Excellence #22: June 2020

Krinkle
📘 Read on Phabricator at
https://phabricator.wikimedia.org/phame/post/view/203/
-------

How’d we do in our strive for operational excellence last month? Read on to
find out!

## 📈   Month in review

* 4 documented incidents in June. [1]
* 37 new production errors were filed and 27 were closed. [2] [3]
* 72 recent production errors still open (up from 68).
* 203 total Wikimedia-prod-error tasks currently open (up from 192). [4]

For more about recent incidents and pending actionables see Wikitech and
Phabricator, at https://wikitech.wikimedia.org/wiki/Incident_documentation
and https://phabricator.wikimedia.org/project/view/4758/

-------

## 📖   Outstanding errors

Breakdown of the errors reported in June that are still open today:

* (Needs owner) / Newsletter extension: Unexpected locking SELECT query.
https://phabricator.wikimedia.org/T253926
* (Needs owner) / FlaggedRevs extension: Unable to submit review of page
due to bad fr_page_id record. https://phabricator.wikimedia.org/T256296
* Editing team / MassMessage extension: Delivery fails due to system user
conflict. https://phabricator.wikimedia.org/T171003
* Parsing team / Parsoid: Pagebundle data unavailable due to a bad UTF-8
string. https://phabricator.wikimedia.org/T236866
* Growth team / Recent changes: Update for ActiveUsers data failing due to
deadlock. https://phabricator.wikimedia.org/T255059
* Growth team / GrowthExperiments: Issue with question display on personal
homepage. https://phabricator.wikimedia.org/T255616
* Language team / Translate extension: Update jobs fail due to invalid
function call. https://phabricator.wikimedia.org/T255669
* Language team / ContentTranslation: Save action fails due to duplicate
insert query. https://phabricator.wikimedia.org/T256230
* Core Platform team / Content handling: Incompatible content type during
content merge/stash. https://phabricator.wikimedia.org/T255700
* Core Platform team / Monolog: API usage logs and error logs sometimes
missing due to socket failure. https://phabricator.wikimedia.org/T255578
* Search Platform team / WikibaseCirrus: Elevated error levels from
EntitySearchElastic warnings. https://phabricator.wikimedia.org/T255658
* Wikidata / API: Generator query fails due to invalid API result format.
https://phabricator.wikimedia.org/T254334
* Wikidata / API: EntityData query emits warning about bad RDF.
https://phabricator.wikimedia.org/T255054
* Wikidata / Repo: Entity relation update jobs fail due to deadlock.
https://phabricator.wikimedia.org/T255706

-------

## 📊   Trends

Take a look at the workboard and look for tasks that could use your help.
→  https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

* July 2019 (5 of 18 tasks left): Two tasks closed.
* August (1 of 14 tasks left): Another task closed, only one remaining! 🚀
* September (5 of 12 tasks left): Two tasks closed.
* October (6 of 12 tasks left), no change.
* November (3 of 5 tasks left): Another task closed.
* December (5 of 9 tasks left), no change.
* January 2020 (5 of 7 tasks lef), no change.
* February (4 of 7 tasks left), no change.
* March (2 of 2 tasks left), no change.
* April (11 of 14 tasks left): Three tasks closed.
* May (11 tasks left): Three tasks closed.
* June: 14 new tasks survived the month of June. ⚠️

At the end of May the number of open production errors over recent months
was 68. Of those, 10 got closed, but with 14 new tasks from June still
open, the total has grown further to 72.

The workboard had 192 open tasks last month, which saw another increase, to
now 203 open tasks (this includes tasks from 2019 and earlier).

-------

## 🎉   Thanks!

Thank you to everyone else who helped by reporting, investigating, or
resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

-------

Footnotes:

[1] Incidents. –
https://wikitech.wikimedia.org/wiki/Incident_documentation#2020

[2] Tasks created. –
https://phabricator.wikimedia.org/maniphest/query/VTpmvaJLYVL1/#R

[3] Tasks closed. –
https://phabricator.wikimedia.org/maniphest/query/qn5yeURqyl3D/#R

[4] Open tasks. –
https://phabricator.wikimedia.org/maniphest/query/Fw3RdXt1Sdxp/#R
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l