MW Train status: Incomplete rollout, blocked on two issues

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

MW Train status: Incomplete rollout, blocked on two issues

Greg Grossmeier-2
Hello,

The 1.32.0-wmf.13 version of MediaWiki is rollout to almost all group0
and group1 wikis, but not group2. In plain wording: It is deployed to
all non-wikipedias (Commons, wiktionaries, etc) except Wikidata.

It was blocked from going to all wikis yesterday due to two issues found
during the week:
* Fatal MWException in Babel: "Language::isValidBuiltInCode must be
  passed a string" - https://phabricator.wikimedia.org/T199941

* Wikidata showing wrong language for page elements -
  https://phabricator.wikimedia.org/T199983

Assuming these issues are resolved before Monday we hope to resume the
deployment of this version Monday during European working hours.

The tracking task for this deployment:
https://phabricator.wikimedia.org/T191059

A handy tool to see which wikis have which version:
https://tools.wmflabs.org/versions/

Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| Release Team Manager            A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Željko Filipin
Hi,

This is the second ever train I am conducting and it has been very
interesting. I've been told that is not unusual. I have good news and bad
news.

Good news is that some problems are resolved. I would like to thank to
everybody that helped.

Bad news is that train is still blocked. :( Blocking tasks are:

- T199941 Fatal MWException in Babel: "Language::isValidBuiltInCode must be
passed a string"
- T199983 Wikidata showing wrong language for page elements
- T200136 Does not work for change a log type drop down when the log type
specified by URL / argument @ 1.32.0-wmf.13 (360f7b5)

I will cut the new branch tomorrow, but I will not be able to deploy it to
group 0 until blockers from last week are resolved. If you can help
resolving the current problems, please do. If you know that somebody can
help, please let them know.

Željko


On Fri, Jul 20, 2018 at 7:19 PM Greg Grossmeier <[hidden email]> wrote:

> Hello,
>
> The 1.32.0-wmf.13 version of MediaWiki is rollout to almost all group0
> and group1 wikis, but not group2. In plain wording: It is deployed to
> all non-wikipedias (Commons, wiktionaries, etc) except Wikidata.
>
> It was blocked from going to all wikis yesterday due to two issues found
> during the week:
> * Fatal MWException in Babel: "Language::isValidBuiltInCode must be
>   passed a string" - https://phabricator.wikimedia.org/T199941
>
> * Wikidata showing wrong language for page elements -
>   https://phabricator.wikimedia.org/T199983
>
> Assuming these issues are resolved before Monday we hope to resume the
> deployment of this version Monday during European working hours.
>
> The tracking task for this deployment:
> https://phabricator.wikimedia.org/T191059
>
> A handy tool to see which wikis have which version:
> https://tools.wmflabs.org/versions/
>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | Release Team Manager            A18D 1138 8E47 FAC8 1C7D |
> _______________________________________________
> Engineering mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Ops] [Engineering] MW Train status: Incomplete rollout, blocked on two issues

C. Scott Ananian
Deployments during the week of wikimania are always tough.  I'm surprised
we even tried to do that this year.
  --scott

On Mon, Jul 23, 2018, 7:23 PM Željko Filipin <[hidden email]> wrote:

> Hi,
>
> This is the second ever train I am conducting and it has been very
> interesting. I've been told that is not unusual. I have good news and bad
> news.
>
> Good news is that some problems are resolved. I would like to thank to
> everybody that helped.
>
> Bad news is that train is still blocked. :( Blocking tasks are:
>
> - T199941 Fatal MWException in Babel: "Language::isValidBuiltInCode must
> be passed a string"
> - T199983 Wikidata showing wrong language for page elements
> - T200136 Does not work for change a log type drop down when the log type
> specified by URL / argument @ 1.32.0-wmf.13 (360f7b5)
>
> I will cut the new branch tomorrow, but I will not be able to deploy it to
> group 0 until blockers from last week are resolved. If you can help
> resolving the current problems, please do. If you know that somebody can
> help, please let them know.
>
> Željko
>
>
> On Fri, Jul 20, 2018 at 7:19 PM Greg Grossmeier <[hidden email]>
> wrote:
>
>> Hello,
>>
>> The 1.32.0-wmf.13 version of MediaWiki is rollout to almost all group0
>> and group1 wikis, but not group2. In plain wording: It is deployed to
>> all non-wikipedias (Commons, wiktionaries, etc) except Wikidata.
>>
>> It was blocked from going to all wikis yesterday due to two issues found
>> during the week:
>> * Fatal MWException in Babel: "Language::isValidBuiltInCode must be
>>   passed a string" - https://phabricator.wikimedia.org/T199941
>>
>> * Wikidata showing wrong language for page elements -
>>   https://phabricator.wikimedia.org/T199983
>>
>> Assuming these issues are resolved before Monday we hope to resume the
>> deployment of this version Monday during European working hours.
>>
>> The tracking task for this deployment:
>> https://phabricator.wikimedia.org/T191059
>>
>> A handy tool to see which wikis have which version:
>> https://tools.wmflabs.org/versions/
>>
>> Greg
>>
>> --
>> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
>> | Release Team Manager            A18D 1138 8E47 FAC8 1C7D |
>> _______________________________________________
>> Engineering mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>
> _______________________________________________
> Ops mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/ops
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Ops] [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Greg Grossmeier-2
<quote name="C. Scott Ananian" date="2018-07-23" time="21:47:25 +0200">
> Deployments during the week of wikimania are always tough.  I'm surprised
> we even tried to do that this year.

There is/was sufficient coverage from both RelEng and SRE so there
wasn't an issue of site outage without available people to respond. IOW,
no one from RelEng nor SRE was attending Wikimania (sadly).

There is the issue of not getting quick feedback and resolution on UBN!
bugs from attendees at the conference, though. (This was obviously
dependent on who was at the conference and only manifested itself in a
couple cases.)

Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| Release Team Manager            A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] [Ops] MW Train status: Incomplete rollout, blocked on two issues

C. Scott Ananian
FWIW, I think the Babel bug was waiting for several days for review from
Niklas, the extension owner, who wasn't at wikimania.  But the folks who
ended up taking action and doing the revert we're at wikimania -- I think
it's a general matter of lack of communication during wikimania, with folks
distracted and assuming someone else is responsible, rather than a specific
"X person is at location Y". Wikimania just disrupts normal communications
and work patterns.
  --scott

(To be clear, I'm not trying to pass blame, I'm just pointing out that the
disruption is not as simple of who is where or whether "enough" people are
at home.)

On Mon, Jul 23, 2018, 4:42 PM Greg Grossmeier <[hidden email]> wrote:

> <quote name="C. Scott Ananian" date="2018-07-23" time="21:47:25 +0200">
> > Deployments during the week of wikimania are always tough.  I'm surprised
> > we even tried to do that this year.
>
> There is/was sufficient coverage from both RelEng and SRE so there
> wasn't an issue of site outage without available people to respond. IOW,
> no one from RelEng nor SRE was attending Wikimania (sadly).
>
> There is the issue of not getting quick feedback and resolution on UBN!
> bugs from attendees at the conference, though. (This was obviously
> dependent on who was at the conference and only manifested itself in a
> couple cases.)
>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | Release Team Manager            A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Engineering mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Wikitech-ambassadors] [Engineering] [Ops] MW Train status: Incomplete rollout, blocked on two issues

Runa Bhattacharjee-2
On Tue, Jul 24, 2018 at 2:59 AM, C. Scott Ananian <[hidden email]>
wrote:

> FWIW, I think the Babel bug was waiting for several days for review from
> Niklas, the extension owner, who wasn't at wikimania.  But the folks who
> ended up taking action and doing the revert we're at wikimania -- I think
> it's a general matter of lack of communication during wikimania, with folks
> distracted and assuming someone else is responsible, rather than a specific
> "X person is at location Y". Wikimania just disrupts normal communications
> and work patterns.
>   --scott
>
> (To be clear, I'm not trying to pass blame, I'm just pointing out that the
> disruption is not as simple of who is where or whether "enough" people are
> at home.)
>

Niklas is on vacation and has marked himself as unavailable on phabricator.
He won't be back until next week. Meanwhile, the Language team did monitor
the conversation and based on that expected the patches to be reverted to
unblock the train.

Thanks
Runa
--
Senior Engineering Manager, Language
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Ops] [Wikitech-ambassadors] [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Željko Filipin
On Tue, Jul 24, 2018 at 10:54 AM Runa Bhattacharjee <
[hidden email]> wrote:

> Meanwhile, the Language team did monitor the conversation and based on
> that expected the patches to be reverted to unblock the train.
>

Hi Runa,

T199941 (Fatal MWException in Babel: "Language::isValidBuiltInCode must be
passed a string") is the only remaining blocker for the train. I see some
activity at the task (commit merges, deployments) but the task is still
open, UBN and a subtask of T191059 (1.32.0-wmf.13 deployment blockers),
making it a blocker.

If the issue is resolved, please resolve the phabricator task. If there is
still some cleanup to do, but it's not blocking the train, please remove
the task from T191059 blockers.

Thanks,

Željko
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Željko Filipin
In reply to this post by Željko Filipin
Hi,

1.32.0-wmf.13 train continues to be "interesting". Trains should not be
interesting. They should be boring.

I would like to thank again everybody that reported problems and worked on
resolving them.

There is currently one blocker:

- T200269 Unable to undelete revision (Fatal error: given Title does not
belong to page ID, RevisionStoreRecord)

I have cut the 1.32.0-wmf.14 branch today, but it's deployment (to groups
0, 1 and 2) is blocked on 1.32.0-wmf.13.

Željko
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Željko Filipin
Good news!

A few minutes ago we have deployed 1.32.0-wmf.13 to all wikis. Logs look OK
to me so far.

A big thank you to everybody that reported problems and helped resolve
them. I want to tank every single one of you, but I'm to tired right now,
so I'll leave that for tomorrow.

For now, I would like to thank Tyler Cipriani and Greg Grossmeier for all
the help with finally getting 1.32.0-wmf.13 everywhere.

Željko
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Pine W
In reply to this post by Željko Filipin
Regarding "1.32.0-wmf.13 train continues to be "interesting". Trains should
not be
interesting. They should be boring.", I am reminded of something that I
read which
is supposedly an ancient Chinese curse: "May you live in interesting times."

It sounds like all ended well, so congratulations on completing your second
train
(my total === 0), and an interesting train at that.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Wikitech-ambassadors] [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Prateek Saxena
In reply to this post by Željko Filipin
As one of the people who made the train "interesting" I'd like to
thank you for taking care of this :)



On Tue, Jul 24, 2018 at 11:20 PM, Željko Filipin <[hidden email]> wrote:

> Good news!
>
> A few minutes ago we have deployed 1.32.0-wmf.13 to all wikis. Logs look OK
> to me so far.
>
> A big thank you to everybody that reported problems and helped resolve them.
> I want to tank every single one of you, but I'm to tired right now, so I'll
> leave that for tomorrow.
>
> For now, I would like to thank Tyler Cipriani and Greg Grossmeier for all
> the help with finally getting 1.32.0-wmf.13 everywhere.
>
> Željko
>
> _______________________________________________
> Wikitech-ambassadors mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-ambassadors
>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] MW Train status: Incomplete rollout, blocked on two issues

Željko Filipin
In reply to this post by Željko Filipin
This will be my final notification about 1.32.0-wmf.13. I have started
working on incident report:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20180717-Train

Any help in completing the report is greatly appreciated.

Thanks,

Željko

On Tue, Jul 24, 2018 at 7:50 PM Željko Filipin <[hidden email]>
wrote:

> Good news!
>
> A few minutes ago we have deployed 1.32.0-wmf.13 to all wikis. Logs look
> OK to me so far.
>
> A big thank you to everybody that reported problems and helped resolve
> them. I want to tank every single one of you, but I'm to tired right now,
> so I'll leave that for tomorrow.
>
> For now, I would like to thank Tyler Cipriani and Greg Grossmeier for all
> the help with finally getting 1.32.0-wmf.13 everywhere.
>
> Željko
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l