Datacenter switchover and switchback

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Datacenter switchover and switchback

Alexandros Kosiaris
Hello everyone,

This is to inform you that there will be a datacenter switchover and
switchback on the next few weeks. The timeline's are

Services: Tuesday, September 11th 2018 14:30 UTC
Media storage/Swift: Tuesday, September 11th 2018 15:00 UTC
Traffic: Tuesday, September 11th 2018 19:00 UTC
MediaWiki: Wednesday, September 12th 2018: 14:00 UTC

Switchback:

Traffic: Wednesday, October 10th 2018 09:00 UTC
MediaWiki: Wednesday, October 10th 2018: 14:00 UTC
Services: Thursday, October 11th 2018 14:30 UTC
Media storage/Swift: Thursday, October 11th 2018 15:00 UTC

For the duration of the switchover (1 month), deployers are kindly
requested to refrain from large db schema changes and avoid deploying
any kind of new feature that requires creation of tables.
There will be a train freeze in the week of Sept 10th and Oct 8th.

The net effect of the switchover and switchback for volunteers is
expected to be some minutes of inability to save an edit. For readers,
everything will be as usual.

The tracking task for interested parties is
https://phabricator.wikimedia.org/T199073

Regards,

--
Alexandros Kosiaris <[hidden email]>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
MA
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

MA
Hello:

>For the duration of the switchover (1 month), deployers are kindly
>requested to refrain from large db schema changes and avoid deploying
>any kind of new feature that requires creation of tables.
>There will be a train freeze in the week of Sept 10th and Oct 8th.

Does this also mean that we should not deploy new extensions to wikis such
as Babel, Echo, ShortUrl or Translate (to mention some examples) where we
also need to create the relevant extension tables on the wiki in question
for them to work?

I guess also no new wiki creations will be allowed in that timespan?

Thanks for the clarification.

Regards.
--
M. A.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Jaime Crespo
Let me explain the rationale of the bellow request for clarification:

On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:

> Hello:
>
> >For the duration of the switchover (1 month), deployers are kindly
> >requested to refrain from large db schema changes and avoid deploying
> >any kind of new feature that requires creation of tables.
> >There will be a train freeze in the week of Sept 10th and Oct 8th.


During the failover, some schema changes will be finalized on the current
active datacenter (plus some major server and network maintenance may be
done)- our request is mostly to refrain from quickly enabling those large
new unlocked features (e.g. the ongoing comment refactoring, actor/user
refactoring, Multi Content Revision, JADE, major wikidata or structured
comons structure changes, new extensions not ever deployed to the cluster,
etc.) at the same time than the ongoing maintenance to reduce variables of
things that can go bad- enabling those features may be unblocked during the
switchover time, but we ask you to hold until being back on the current
active datacenter. Basically, ask yourself if you are enabling a large new
core feature or want to start a heavy-write maintenance script and there is
a chance you will need DBA/system support. Sadly, we had some instances of
this happening last year and we want to explicitly discourage this during
these 2 weeks.

In own my opinion, enabling existing features on smaller projects (size
here is in amount of server resources, not that they are less important) is
equivalent to a swat change, and I am not against it happening. I would ask
contributors to use their best judgement on every case, and ask people on
the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
My plea is to not enable major structural changes during that time may
affect thousands of edits per minute. Swat-like changes and "boring" :-)
trains are ok.

For new wiki creations I would prefer if those were delayed but CC #DBA s
on the phabricator task to check with us.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Derk-Jan Hartman
While I think these regular switches are a very good idea, from an outside
perspective I do have to question a process that puts a significant plug in
the velocity of various teams working on major projects (esp. in a time of
year that could probably be seen as one of the most productive). What are
plans to reduce the disruption of this exercise in the future ?

DJ

On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]> wrote:

> Let me explain the rationale of the bellow request for clarification:
>
> On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
>
> > Hello:
> >
> > >For the duration of the switchover (1 month), deployers are kindly
> > >requested to refrain from large db schema changes and avoid deploying
> > >any kind of new feature that requires creation of tables.
> > >There will be a train freeze in the week of Sept 10th and Oct 8th.
>
>
> During the failover, some schema changes will be finalized on the current
> active datacenter (plus some major server and network maintenance may be
> done)- our request is mostly to refrain from quickly enabling those large
> new unlocked features (e.g. the ongoing comment refactoring, actor/user
> refactoring, Multi Content Revision, JADE, major wikidata or structured
> comons structure changes, new extensions not ever deployed to the cluster,
> etc.) at the same time than the ongoing maintenance to reduce variables of
> things that can go bad- enabling those features may be unblocked during the
> switchover time, but we ask you to hold until being back on the current
> active datacenter. Basically, ask yourself if you are enabling a large new
> core feature or want to start a heavy-write maintenance script and there is
> a chance you will need DBA/system support. Sadly, we had some instances of
> this happening last year and we want to explicitly discourage this during
> these 2 weeks.
>
> In own my opinion, enabling existing features on smaller projects (size
> here is in amount of server resources, not that they are less important) is
> equivalent to a swat change, and I am not against it happening. I would ask
> contributors to use their best judgement on every case, and ask people on
> the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> My plea is to not enable major structural changes during that time may
> affect thousands of edits per minute. Swat-like changes and "boring" :-)
> trains are ok.
>
> For new wiki creations I would prefer if those were delayed but CC #DBA s
> on the phabricator task to check with us.
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Pine W
+1 to DJ's question about timing. Also, one might wish to be mindful of the number of recent trains that were supposed to be boring but involved interesting surprises; this makes me wonder whether trains that one thinks will be boring are actually OK in this circumstance even if they turn out to be "interesting".

Pine
( https://meta.wikimedia.org/wiki/User:Pine )



-------- Original message --------From: Derk-Jan Hartman <[hidden email]> Date: 8/30/18  2:54 AM  (GMT-08:00) To: Wikimedia developers <[hidden email]> Subject: Re: [Wikitech-l] Datacenter switchover and switchback
While I think these regular switches are a very good idea, from an outside
perspective I do have to question a process that puts a significant plug in
the velocity of various teams working on major projects (esp. in a time of
year that could probably be seen as one of the most productive). What are
plans to reduce the disruption of this exercise in the future ?

DJ

On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]> wrote:

> Let me explain the rationale of the bellow request for clarification:
>
> On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
>
> > Hello:
> >
> > >For the duration of the switchover (1 month), deployers are kindly
> > >requested to refrain from large db schema changes and avoid deploying
> > >any kind of new feature that requires creation of tables.
> > >There will be a train freeze in the week of Sept 10th and Oct 8th.
>
>
> During the failover, some schema changes will be finalized on the current
> active datacenter (plus some major server and network maintenance may be
> done)- our request is mostly to refrain from quickly enabling those large
> new unlocked features (e.g. the ongoing comment refactoring, actor/user
> refactoring, Multi Content Revision, JADE, major wikidata or structured
> comons structure changes, new extensions not ever deployed to the cluster,
> etc.) at the same time than the ongoing maintenance to reduce variables of
> things that can go bad- enabling those features may be unblocked during the
> switchover time, but we ask you to hold until being back on the current
> active datacenter. Basically, ask yourself if you are enabling a large new
> core feature or want to start a heavy-write maintenance script and there is
> a chance you will need DBA/system support. Sadly, we had some instances of
> this happening last year and we want to explicitly discourage this during
> these 2 weeks.
>
> In own my opinion, enabling existing features on smaller projects (size
> here is in amount of server resources, not that they are less important) is
> equivalent to a swat change, and I am not against it happening. I would ask
> contributors to use their best judgement on every case, and ask people on
> the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> My plea is to not enable major structural changes during that time may
> affect thousands of edits per minute. Swat-like changes and "boring" :-)
> trains are ok.
>
> For new wiki creations I would prefer if those were delayed but CC #DBA s
> on the phabricator task to check with us.
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Pine W
A couple of additional points came to mind.

1. Blocking the creation of new wikis sounds like it could be a big deal. I know little about the process for getting new wikis approved and launched, but I hope that the folks who are regularly involved in these processes have been advised of the situation.

2. In my previous email I may have revealed my level of ignorance about the deployment train. Hopefully I didn't come across as presuming to "sound smart" about that subject. I know that I am in the presence of experts on that subject.

Regards,

Pine
( https://meta.wikimedia.org/wiki/User:Pine )



-------- Original message --------From: Pine W <[hidden email]> Date: 8/30/18  7:57 AM  (GMT-08:00) To: Wikimedia developers <[hidden email]> Subject: Re: [Wikitech-l] Datacenter switchover and switchback
+1 to DJ's question about timing. Also, one might wish to be mindful of the number of recent trains that were supposed to be boring but involved interesting surprises; this makes me wonder whether trains that one thinks will be boring are actually OK in this circumstance even if they turn out to be "interesting".

Pine
( https://meta.wikimedia.org/wiki/User:Pine )



-------- Original message --------From: Derk-Jan Hartman <[hidden email]> Date: 8/30/18  2:54 AM  (GMT-08:00) To: Wikimedia developers <[hidden email]> Subject: Re: [Wikitech-l] Datacenter switchover and switchback
While I think these regular switches are a very good idea, from an outside
perspective I do have to question a process that puts a significant plug in
the velocity of various teams working on major projects (esp. in a time of
year that could probably be seen as one of the most productive). What are
plans to reduce the disruption of this exercise in the future ?

DJ

On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]> wrote:

> Let me explain the rationale of the bellow request for clarification:
>
> On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
>
> > Hello:
> >
> > >For the duration of the switchover (1 month), deployers are kindly
> > >requested to refrain from large db schema changes and avoid deploying
> > >any kind of new feature that requires creation of tables.
> > >There will be a train freeze in the week of Sept 10th and Oct 8th.
>
>
> During the failover, some schema changes will be finalized on the current
> active datacenter (plus some major server and network maintenance may be
> done)- our request is mostly to refrain from quickly enabling those large
> new unlocked features (e.g. the ongoing comment refactoring, actor/user
> refactoring, Multi Content Revision, JADE, major wikidata or structured
> comons structure changes, new extensions not ever deployed to the cluster,
> etc.) at the same time than the ongoing maintenance to reduce variables of
> things that can go bad- enabling those features may be unblocked during the
> switchover time, but we ask you to hold until being back on the current
> active datacenter. Basically, ask yourself if you are enabling a large new
> core feature or want to start a heavy-write maintenance script and there is
> a chance you will need DBA/system support. Sadly, we had some instances of
> this happening last year and we want to explicitly discourage this during
> these 2 weeks.
>
> In own my opinion, enabling existing features on smaller projects (size
> here is in amount of server resources, not that they are less important) is
> equivalent to a swat change, and I am not against it happening. I would ask
> contributors to use their best judgement on every case, and ask people on
> the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> My plea is to not enable major structural changes during that time may
> affect thousands of edits per minute. Swat-like changes and "boring" :-)
> trains are ok.
>
> For new wiki creations I would prefer if those were delayed but CC #DBA s
> on the phabricator task to check with us.
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Alex Monk
Wiki creation relies on experienced deployers and ops, I would expect they
all know.

On Thu, 30 Aug 2018, 16:49 Pine W, <[hidden email]> wrote:

> A couple of additional points came to mind.
>
> 1. Blocking the creation of new wikis sounds like it could be a big deal.
> I know little about the process for getting new wikis approved and
> launched, but I hope that the folks who are regularly involved in these
> processes have been advised of the situation.
>
> 2. In my previous email I may have revealed my level of ignorance about
> the deployment train. Hopefully I didn't come across as presuming to "sound
> smart" about that subject. I know that I am in the presence of experts on
> that subject.
>
> Regards,
>
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
>
>
> -------- Original message --------From: Pine W <[hidden email]>
> Date: 8/30/18  7:57 AM  (GMT-08:00) To: Wikimedia developers <
> [hidden email]> Subject: Re: [Wikitech-l] Datacenter
> switchover and switchback
> +1 to DJ's question about timing. Also, one might wish to be mindful of
> the number of recent trains that were supposed to be boring but involved
> interesting surprises; this makes me wonder whether trains that one thinks
> will be boring are actually OK in this circumstance even if they turn out
> to be "interesting".
>
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
>
>
> -------- Original message --------From: Derk-Jan Hartman <
> [hidden email]> Date: 8/30/18  2:54 AM  (GMT-08:00) To:
> Wikimedia developers <[hidden email]> Subject: Re:
> [Wikitech-l] Datacenter switchover and switchback
> While I think these regular switches are a very good idea, from an outside
> perspective I do have to question a process that puts a significant plug in
> the velocity of various teams working on major projects (esp. in a time of
> year that could probably be seen as one of the most productive). What are
> plans to reduce the disruption of this exercise in the future ?
>
> DJ
>
> On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]>
> wrote:
>
> > Let me explain the rationale of the bellow request for clarification:
> >
> > On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
> >
> > > Hello:
> > >
> > > >For the duration of the switchover (1 month), deployers are kindly
> > > >requested to refrain from large db schema changes and avoid deploying
> > > >any kind of new feature that requires creation of tables.
> > > >There will be a train freeze in the week of Sept 10th and Oct 8th.
> >
> >
> > During the failover, some schema changes will be finalized on the current
> > active datacenter (plus some major server and network maintenance may be
> > done)- our request is mostly to refrain from quickly enabling those large
> > new unlocked features (e.g. the ongoing comment refactoring, actor/user
> > refactoring, Multi Content Revision, JADE, major wikidata or structured
> > comons structure changes, new extensions not ever deployed to the
> cluster,
> > etc.) at the same time than the ongoing maintenance to reduce variables
> of
> > things that can go bad- enabling those features may be unblocked during
> the
> > switchover time, but we ask you to hold until being back on the current
> > active datacenter. Basically, ask yourself if you are enabling a large
> new
> > core feature or want to start a heavy-write maintenance script and there
> is
> > a chance you will need DBA/system support. Sadly, we had some instances
> of
> > this happening last year and we want to explicitly discourage this during
> > these 2 weeks.
> >
> > In own my opinion, enabling existing features on smaller projects (size
> > here is in amount of server resources, not that they are less important)
> is
> > equivalent to a swat change, and I am not against it happening. I would
> ask
> > contributors to use their best judgement on every case, and ask people on
> > the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> > My plea is to not enable major structural changes during that time may
> > affect thousands of edits per minute. Swat-like changes and "boring" :-)
> > trains are ok.
> >
> > For new wiki creations I would prefer if those were delayed but CC #DBA s
> > on the phabricator task to check with us.
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Alexandros Kosiaris
In reply to this post by Derk-Jan Hartman
On Thu, Aug 30, 2018 at 12:55 PM Derk-Jan Hartman
<[hidden email]> wrote:
>
> While I think these regular switches are a very good idea, from an outside
> perspective I do have to question a process that puts a significant plug in
> the velocity of various teams working on major projects (esp. in a time of
> year that could probably be seen as one of the most productive).

That is absolutely true.
> What are
> plans to reduce the disruption of this exercise in the future ?

There are indeed plans about making this easier, more automated and
shorter in duration. Just to name a few, mediawiki now no longer
requires deployments to switch datacenters but rather relies on a
state key in the etcd database, there is a library + cookbooks to
automate the currently automateable steps, there is work to update the
documentation and make it update to date, accurate and more usable by
more people and at shorter notices and so on. That being said, it's
never gonna be free, but the toil is significant to make us want to
reduce and that what we aim for.


>
> DJ
>
> On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]> wrote:
>
> > Let me explain the rationale of the bellow request for clarification:
> >
> > On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
> >
> > > Hello:
> > >
> > > >For the duration of the switchover (1 month), deployers are kindly
> > > >requested to refrain from large db schema changes and avoid deploying
> > > >any kind of new feature that requires creation of tables.
> > > >There will be a train freeze in the week of Sept 10th and Oct 8th.
> >
> >
> > During the failover, some schema changes will be finalized on the current
> > active datacenter (plus some major server and network maintenance may be
> > done)- our request is mostly to refrain from quickly enabling those large
> > new unlocked features (e.g. the ongoing comment refactoring, actor/user
> > refactoring, Multi Content Revision, JADE, major wikidata or structured
> > comons structure changes, new extensions not ever deployed to the cluster,
> > etc.) at the same time than the ongoing maintenance to reduce variables of
> > things that can go bad- enabling those features may be unblocked during the
> > switchover time, but we ask you to hold until being back on the current
> > active datacenter. Basically, ask yourself if you are enabling a large new
> > core feature or want to start a heavy-write maintenance script and there is
> > a chance you will need DBA/system support. Sadly, we had some instances of
> > this happening last year and we want to explicitly discourage this during
> > these 2 weeks.
> >
> > In own my opinion, enabling existing features on smaller projects (size
> > here is in amount of server resources, not that they are less important) is
> > equivalent to a swat change, and I am not against it happening. I would ask
> > contributors to use their best judgement on every case, and ask people on
> > the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> > My plea is to not enable major structural changes during that time may
> > affect thousands of edits per minute. Swat-like changes and "boring" :-)
> > trains are ok.
> >
> > For new wiki creations I would prefer if those were delayed but CC #DBA s
> > on the phabricator task to check with us.
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Alexandros Kosiaris <[hidden email]>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Alexandros Kosiaris
In reply to this post by Alexandros Kosiaris
Reminder: The switchback is next week.
On Wed, Aug 29, 2018 at 8:28 PM Alexandros Kosiaris
<[hidden email]> wrote:

>
> Hello everyone,
>
> This is to inform you that there will be a datacenter switchover and
> switchback on the next few weeks. The timeline's are
>
> Services: Tuesday, September 11th 2018 14:30 UTC
> Media storage/Swift: Tuesday, September 11th 2018 15:00 UTC
> Traffic: Tuesday, September 11th 2018 19:00 UTC
> MediaWiki: Wednesday, September 12th 2018: 14:00 UTC
>
> Switchback:
>
> Traffic: Wednesday, October 10th 2018 09:00 UTC
> MediaWiki: Wednesday, October 10th 2018: 14:00 UTC
> Services: Thursday, October 11th 2018 14:30 UTC
> Media storage/Swift: Thursday, October 11th 2018 15:00 UTC
>
> For the duration of the switchover (1 month), deployers are kindly
> requested to refrain from large db schema changes and avoid deploying
> any kind of new feature that requires creation of tables.
> There will be a train freeze in the week of Sept 10th and Oct 8th.
>
> The net effect of the switchover and switchback for volunteers is
> expected to be some minutes of inability to save an edit. For readers,
> everything will be as usual.
>
> The tracking task for interested parties is
> https://phabricator.wikimedia.org/T199073
>
> Regards,
>
> --
> Alexandros Kosiaris <[hidden email]>



--
Alexandros Kosiaris <[hidden email]>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

C. Scott Ananian
In reply to this post by Pine W
Oct 8 seems to be a particularly bad time to freeze the train given that we
are forking for the MW 1.32 release on Oct 15, and a lot of folks have
last-minute things they want to get into the release (eg deprecations, etc).
  --scott

On Thu, Aug 30, 2018 at 10:57 AM Pine W <[hidden email]> wrote:

> +1 to DJ's question about timing. Also, one might wish to be mindful of
> the number of recent trains that were supposed to be boring but involved
> interesting surprises; this makes me wonder whether trains that one thinks
> will be boring are actually OK in this circumstance even if they turn out
> to be "interesting".
>
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
>
>
> -------- Original message --------From: Derk-Jan Hartman <
> [hidden email]> Date: 8/30/18  2:54 AM  (GMT-08:00) To:
> Wikimedia developers <[hidden email]> Subject: Re:
> [Wikitech-l] Datacenter switchover and switchback
> While I think these regular switches are a very good idea, from an outside
> perspective I do have to question a process that puts a significant plug in
> the velocity of various teams working on major projects (esp. in a time of
> year that could probably be seen as one of the most productive). What are
> plans to reduce the disruption of this exercise in the future ?
>
> DJ
>
> On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]>
> wrote:
>
> > Let me explain the rationale of the bellow request for clarification:
> >
> > On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
> >
> > > Hello:
> > >
> > > >For the duration of the switchover (1 month), deployers are kindly
> > > >requested to refrain from large db schema changes and avoid deploying
> > > >any kind of new feature that requires creation of tables.
> > > >There will be a train freeze in the week of Sept 10th and Oct 8th.
> >
> >
> > During the failover, some schema changes will be finalized on the current
> > active datacenter (plus some major server and network maintenance may be
> > done)- our request is mostly to refrain from quickly enabling those large
> > new unlocked features (e.g. the ongoing comment refactoring, actor/user
> > refactoring, Multi Content Revision, JADE, major wikidata or structured
> > comons structure changes, new extensions not ever deployed to the
> cluster,
> > etc.) at the same time than the ongoing maintenance to reduce variables
> of
> > things that can go bad- enabling those features may be unblocked during
> the
> > switchover time, but we ask you to hold until being back on the current
> > active datacenter. Basically, ask yourself if you are enabling a large
> new
> > core feature or want to start a heavy-write maintenance script and there
> is
> > a chance you will need DBA/system support. Sadly, we had some instances
> of
> > this happening last year and we want to explicitly discourage this during
> > these 2 weeks.
> >
> > In own my opinion, enabling existing features on smaller projects (size
> > here is in amount of server resources, not that they are less important)
> is
> > equivalent to a swat change, and I am not against it happening. I would
> ask
> > contributors to use their best judgement on every case, and ask people on
> > the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> > My plea is to not enable major structural changes during that time may
> > affect thousands of edits per minute. Swat-like changes and "boring" :-)
> > trains are ok.
> >
> > For new wiki creations I would prefer if those were delayed but CC #DBA s
> > on the phabricator task to check with us.
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
(http://cscott.net)
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Datacenter switchover and switchback

Alexandros Kosiaris
I am sorry to hear that. It looks like something that we will have to
take into account for the next switchovers. That being said, we had
deliberations across the involved teams months ago to come up with
those exact dates and have been communicating them via at least SoS
since 2018-08-01.

I am curious about something though. How does the deployment train
(cause that's what we are talking about) impact the software release
exactly ?
On Wed, Oct 3, 2018 at 6:19 PM C. Scott Ananian <[hidden email]> wrote:

>
> Oct 8 seems to be a particularly bad time to freeze the train given that we
> are forking for the MW 1.32 release on Oct 15, and a lot of folks have
> last-minute things they want to get into the release (eg deprecations, etc).
>   --scott
>
> On Thu, Aug 30, 2018 at 10:57 AM Pine W <[hidden email]> wrote:
>
> > +1 to DJ's question about timing. Also, one might wish to be mindful of
> > the number of recent trains that were supposed to be boring but involved
> > interesting surprises; this makes me wonder whether trains that one thinks
> > will be boring are actually OK in this circumstance even if they turn out
> > to be "interesting".
> >
> > Pine
> > ( https://meta.wikimedia.org/wiki/User:Pine )
> >
> >
> >
> > -------- Original message --------From: Derk-Jan Hartman <
> > [hidden email]> Date: 8/30/18  2:54 AM  (GMT-08:00) To:
> > Wikimedia developers <[hidden email]> Subject: Re:
> > [Wikitech-l] Datacenter switchover and switchback
> > While I think these regular switches are a very good idea, from an outside
> > perspective I do have to question a process that puts a significant plug in
> > the velocity of various teams working on major projects (esp. in a time of
> > year that could probably be seen as one of the most productive). What are
> > plans to reduce the disruption of this exercise in the future ?
> >
> > DJ
> >
> > On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo <[hidden email]>
> > wrote:
> >
> > > Let me explain the rationale of the bellow request for clarification:
> > >
> > > On Wed, Aug 29, 2018 at 11:30 PM MA <[hidden email]> wrote:
> > >
> > > > Hello:
> > > >
> > > > >For the duration of the switchover (1 month), deployers are kindly
> > > > >requested to refrain from large db schema changes and avoid deploying
> > > > >any kind of new feature that requires creation of tables.
> > > > >There will be a train freeze in the week of Sept 10th and Oct 8th.
> > >
> > >
> > > During the failover, some schema changes will be finalized on the current
> > > active datacenter (plus some major server and network maintenance may be
> > > done)- our request is mostly to refrain from quickly enabling those large
> > > new unlocked features (e.g. the ongoing comment refactoring, actor/user
> > > refactoring, Multi Content Revision, JADE, major wikidata or structured
> > > comons structure changes, new extensions not ever deployed to the
> > cluster,
> > > etc.) at the same time than the ongoing maintenance to reduce variables
> > of
> > > things that can go bad- enabling those features may be unblocked during
> > the
> > > switchover time, but we ask you to hold until being back on the current
> > > active datacenter. Basically, ask yourself if you are enabling a large
> > new
> > > core feature or want to start a heavy-write maintenance script and there
> > is
> > > a chance you will need DBA/system support. Sadly, we had some instances
> > of
> > > this happening last year and we want to explicitly discourage this during
> > > these 2 weeks.
> > >
> > > In own my opinion, enabling existing features on smaller projects (size
> > > here is in amount of server resources, not that they are less important)
> > is
> > > equivalent to a swat change, and I am not against it happening. I would
> > ask
> > > contributors to use their best judgement on every case, and ask people on
> > > the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> > > My plea is to not enable major structural changes during that time may
> > > affect thousands of edits per minute. Swat-like changes and "boring" :-)
> > > trains are ok.
> > >
> > > For new wiki creations I would prefer if those were delayed but CC #DBA s
> > > on the phabricator task to check with us.
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> (http://cscott.net)
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Alexandros Kosiaris <[hidden email]>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l