[Train] 1.34.0-wmf.22 status update

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Train] 1.34.0-wmf.22 status update

Antoine Musso-3
Hello,

Deployment of MediaWiki 1.34.0-wmf.22 on the Wikimedia cluster is
currently blocked. It is blocked on a php7.2 bug that has yet to be
tracked and determined.


Immediately after deploying to group1 (which has commons and wikidata),
I noticed an infamous error in the logs:

  PHP Notice: Undefined index:"

Which sounds similar to known errors with php7.2 (T229433, T231089 among
others) but this time it hits LBFactoryMulti.php which is for the
database load balancing. So that sounds scary and I thus decided to
block the train.

The notice happens at:

  $groupLoads[ILoadBalancer::GROUP_GENERIC]

The GROUP_GENERIC constant is an empty string. Brad "anomie" Jorsch
quickly found out the code always set that key!

https://phabricator.wikimedia.org/T232613#5485181

So what is left is a edge case issue in php 7.2, maybe in the opcache or
something even scarier.

Thanks to Tim Starling, we have a solution to issue a core dump of php
when that issue happens. There is already available on the cluster, but
it has to be investigated which is not that trivial.


--
Antoine Musso



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Train] 1.34.0-wmf.22 status update

Antoine Musso-3
On 12/09/2019 23:42, Antoine Musso wrote:

> Hello,
>
> Deployment of MediaWiki 1.34.0-wmf.22 on the Wikimedia cluster is
> currently blocked. It is blocked on a php7.2 bug that has yet to be
> tracked and determined.
>
>
> Immediately after deploying to group1 (which has commons and wikidata),
> I noticed an infamous error in the logs:
>
>   PHP Notice: Undefined index:"
>
> Which sounds similar to known errors with php7.2 (T229433, T231089 among
> others) but this time it hits LBFactoryMulti.php which is for the
> database load balancing. So that sounds scary and I thus decided to
> block the train.
>
> The notice happens at:
>
>   $groupLoads[ILoadBalancer::GROUP_GENERIC]
>
> The GROUP_GENERIC constant is an empty string. Brad "anomie" Jorsch
> quickly found out the code always set that key!
>
> https://phabricator.wikimedia.org/T232613#5485181
>
> So what is left is a edge case issue in php 7.2, maybe in the opcache or
> something even scarier.
>
> Thanks to Tim Starling, we have a solution to issue a core dump of php
> when that issue happens. There is already available on the cluster, but
> it has to be investigated which is not that trivial.

Hello,

Eventually Tim Starling and Giuseppe Lavagetto continued the debugging
over the week-end (thank you!).  The issue came from the PHP Memcached
extension:
https://phabricator.wikimedia.org/T232613#5494695

The fix has been deployed on the Wikimedia Cluster at 12:00 UTC and the
errors are gone.

I am thus processing promoting 1.34.0-wmf.22 to the rest of the wikis.


Thank you Tim and Giuseppe!

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l