Upgrade of 1.28.0-wmf.19 to group 1 is on hold

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade of 1.28.0-wmf.19 to group 1 is on hold

Antoine Musso-3
Hello,

We are holding the deployment of MediaWiki version 1.28.0-wmf.19 due to
a couple of bugs that have surfaced.

The first one is that renaming a user was blocked [T145596]. Reported by
K6ka, triaged by MarcoAurelio.  The issue is fixed now thanks to Aaron
Schulz and Kunal Mehta.

The second blocker is way nastier. I have pushed the upgrade to group1.
matanya (a long volunteer with a lot of technical patches) immediately
reported the infoboxes on the Hebrew Wiki were on the wrong side, which
prompted a rollback.  [T145673].

Timo Tijhof and Kunal Mehta have found the root cause.  I can speak for
them as to who/when we will get a solution.


For now. I am holding the train.  Will reassess tomorrow and ideally
push group1 at 19:00 UTC then follow with group2 at 20:00UTC.


Stay tuned!



Up-to-date status:
https://tools.wmflabs.org/versions/

MW-1.28.0-wmf.19 deployment blockers
https://phabricator.wikimedia.org/T143328


[T145596] Renames getting stuck on mediawiki.org (Sept 13, 2016)
    https://phabricator.wikimedia.org/T145596

[T145673] 1.28.0-wmf.19 broke template RTL placement
https://phabricator.wikimedia.org/T145673

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Wikis back to 1.28.0-wmf.18 (was: Re: Upgrade of 1.28.0-wmf.19 to group 1 is on hold)

Antoine Musso-3
On 14/09/16 23:05, Antoine Musso wrote:
> <snip>
> For now. I am holding the train.  Will reassess tomorrow and ideally
> push group1 at 19:00 UTC then follow with group2 at 20:00UTC.
..
>
> Up-to-date status:
> https://tools.wmflabs.org/versions/
>
> MW-1.28.0-wmf.19 deployment blockers
> https://phabricator.wikimedia.org/T143328

Hello,

I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got
noticed that Wikidata was no more able to dispatch updates to the wikis
which is tracked in:

https://phabricator.wikimedia.org/T145819

This Friday, I was busy debugging the issue and did not notice a task
about account creation being broken:
  https://phabricator.wikimedia.org/T145839

As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.

Account creation has been impossible from Thursday 19:10 UTC until
Friday 12:50 UTC.   Please pass the word around as needed.

My deep apologizes for not having found out earlier the impact was so
important.

I will write an incident report this afternoon with actionables.

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikis back to 1.28.0-wmf.18

Antoine Musso-3
On 16/09/16 15:29, Antoine Musso wrote:

>
> I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got
> noticed that Wikidata was no more able to dispatch updates to the wikis
> which is tracked in:
>
> https://phabricator.wikimedia.org/T145819
>
> This Friday, I was busy debugging the issue and did not notice a task
> about account creation being broken:
>  https://phabricator.wikimedia.org/T145839
>
> As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.
>
> Account creation has been impossible from Thursday 19:10 UTC until
> Friday 12:50 UTC.   Please pass the word around as needed.
>
> My deep apologizes for not having found out earlier the impact was so
> important.
>
> I will write an incident report this afternoon with actionables.

A preliminary incident report that has not been peer reviewed is
available at:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki

For the time being, all wikis are staying on 1.28.0-wmf.18.

We will discuss the best course of action on Monday.  Most probably get
a proper fix then push wmf.19 and shift wmf.20 to later.

--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikis back to 1.28.0-wmf.18

Pine W
Thank you for trying to untangle this web. I'm going to comment on the
incident report talk page.

Pine

On Fri, Sep 16, 2016 at 7:57 AM, Antoine Musso <[hidden email]> wrote:

> On 16/09/16 15:29, Antoine Musso wrote:
>
>>
>> I have pushed 1.28.0-wmf.19 on thursday at 19:10 UTC. It quickly got
>> noticed that Wikidata was no more able to dispatch updates to the wikis
>> which is tracked in:
>>
>> https://phabricator.wikimedia.org/T145819
>>
>> This Friday, I was busy debugging the issue and did not notice a task
>> about account creation being broken:
>>  https://phabricator.wikimedia.org/T145839
>>
>> As soon as I seen that, I have reverted to 1.28.0-wmf.18 at 12:50 UTC.
>>
>> Account creation has been impossible from Thursday 19:10 UTC until
>> Friday 12:50 UTC.   Please pass the word around as needed.
>>
>> My deep apologizes for not having found out earlier the impact was so
>> important.
>>
>> I will write an incident report this afternoon with actionables.
>>
>
> A preliminary incident report that has not been peer reviewed is available
> at:
>
> https://wikitech.wikimedia.org/wiki/Incident_documentation/
> 20160915-MediaWiki
>
> For the time being, all wikis are staying on 1.28.0-wmf.18.
>
> We will discuss the best course of action on Monday.  Most probably get a
> proper fix then push wmf.19 and shift wmf.20 to later.
>
> --
> Antoine "hashar" Musso
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Wikis paused at 1.28.0-wmf.18 (was: Re: Upgrade of 1.28.0-wmf.19 to group 1 is on hold)

Tyler Cipriani
In reply to this post by Antoine Musso-3
tl;dr: All wikis are staying at 1.28.0-wmf.18 for now

Last week all wikis were rolled back to MediaWiki version 1.28.0-wmf.18
due to several problems that were spotted on Friday (2016-09-16)[0][1].

The problems with wmf.19 seemed resolved by Monday (2016-09-19). The
plan was to roll wmf.19 out to all wikis yesterday afternoon and continue
with the wmf.20 branch-cut today as scheduled.

Late yesterday a large performance regression was discovered in
wmf.18[2].

We've paused the rollout of wmf.19 and the branching of wmf.20 to allow
time to investigate the nature of this performance regression.

After there is a better understanding of the performance regression, we
will reevaluate our plans for branching and rollout of wmf.19 and wmf.20.

-- Tyler

[0]. https://phabricator.wikimedia.org/T111441
[1]. https://phabricator.wikimedia.org/T145819
[2]. https://phabricator.wikimedia.org/T146099

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikis paused at 1.28.0-wmf.18 (was: Re: Upgrade of 1.28.0-wmf.19 to group 1 is on hold)

Roan Kattouw-2
What are your plans for when you'll deploy wmf.19, branch wmf.20 and deploy
wmf.20?

I ask because I have a patch lined up that I wanted to merge today right
after the wmf.20 branch cut, because I want it to spend the full week on
beta labs before it goes to production. I can merge that patch today and
have it go to beta, and have it go into wmf.20 once it gets cut, but I'd
only be willing to do that if I had a guarantee that wmf.20 would not be
cut before early next week. Because it's currently unknown when wmf.20 will
be cut, I don't know what's safe to do.

Unrelatedly, I also have a patch merged in master that fixes a regression
in wmf.19. Since wmf.19 is not deployed right now, the regression doesn't
affect production, but if wmf.19 is ever going to be deployed, then the fix
needs to be cherry-picked into it. Do you know if wmf.19 is going to be
deployed, or is it going to be skipped? Since wmf.19 is not deployed right
now, should I submit a cherry-pick to it for SWAT today so that it gets
unbroken in the event it ever gets deployed?

I understand that you are investigating and will reevaluate after, but it
would be nice to have these questions answered quickly so we have some
sense of what is and isn't going to happen, and aren't blocked by this
uncertainty for much longer,

On Tue, Sep 20, 2016 at 12:27 PM, Tyler Cipriani <[hidden email]>
wrote:

> tl;dr: All wikis are staying at 1.28.0-wmf.18 for now
>
> Last week all wikis were rolled back to MediaWiki version 1.28.0-wmf.18
> due to several problems that were spotted on Friday (2016-09-16)[0][1].
>
> The problems with wmf.19 seemed resolved by Monday (2016-09-19). The
> plan was to roll wmf.19 out to all wikis yesterday afternoon and continue
> with the wmf.20 branch-cut today as scheduled.
>
> Late yesterday a large performance regression was discovered in
> wmf.18[2].
>
> We've paused the rollout of wmf.19 and the branching of wmf.20 to allow
> time to investigate the nature of this performance regression.
>
> After there is a better understanding of the performance regression, we
> will reevaluate our plans for branching and rollout of wmf.19 and wmf.20.
>
> -- Tyler
>
> [0]. https://phabricator.wikimedia.org/T111441
> [1]. https://phabricator.wikimedia.org/T145819
> [2]. https://phabricator.wikimedia.org/T146099
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikis paused at 1.28.0-wmf.18 (was: Re: Upgrade of 1.28.0-wmf.19 to group 1 is on hold)

Tyler Cipriani
In reply to this post by Tyler Cipriani
I have cut 1.28.0-wmf.20[0] for MediaWiki and extensions as the branch
cut was blocking merges to master for developers.

The state of deployed code has not changed – all wikis are running the
1.28.0-wmf.18 branch of MediaWiki and extensions.

Plans for moving forward are still being discussed on Phabricator[1].

-- Tyler

[0]. <https://phabricator.wikimedia.org/diffusion/MW/browse/wmf%252F1.28.0-wmf.20/>
[1]. <https://phabricator.wikimedia.org/T144644>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Group0 on wmf.20, all to wmf.20 tomorrow (was: Re: Upgrade of 1.28.0-wmf.19 to group 1 is on hold)

Tyler Cipriani
In reply to this post by Antoine Musso-3
Group0 wikis (mediawikiwiki, test2wiki, testwiki, testwikidatawiki, and
zerowiki) are running version 1.28.0-wmf.20 of MediaWiki and extensions.
All other wikis are running 1.28.0-wmf.18.

Tomorrow there will be a shortened train schedule in the normal train
deployment window during which 1.28.0-wmf.20 will be pushed to all
wikis.

Any blockers to this plan are tracked on Phabricator[0].

Thank you for all your help and patience while we get the train
schedule[1] back on track!

-- Tyler

[0]. <https://phabricator.wikimedia.org/T144644>
[1]. <https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l