Quantcast

Fwd: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr 17th/May 1st

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr 17th/May 1st

zppix e



> From: Faidon Liambotis <[hidden email]>
> Date: April 7, 2017 at 8:58:09 AM CDT
> To: Wikimedia developers <[hidden email]>
> Subject: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr 17th/May 1st
> Reply-To: Wikimedia developers <[hidden email]>
>
> Hi all,
>
> You may have heard already that, like last year, we are planning to
> switch our active datacenter from eqiad to codfw in the week of April
> 17th and back to eqiad two weeks later, on the week of May 1st. We do
> this periodically in order to exercise our ability to run from the
> backup site in case of a disaster, as well as our ability to switch
> seamlessly to it with little user impact.
>
> Switching will be a gradual, multi-step process, the most visible step
> of which will be the switch of MediaWiki application servers and
> associated data stores. This will happen on April 19th (eqiad->codfw)
> and May 3rd (codfw->eqiad), both at 14:00 UTC. During those windows, the
> sites will be placed into read-only mode, for a period that we estimate
> to last approximately 20 to 30 minutes.
>
> Furthermore, the deployment train will freeze for the weeks of April
> 17th and May 1st[1], but operate normally on the week of April 24th, in
> order to exercise our ability to deploy code while operating from the
> backup datacenter.
>
> 1: https://wikitech.wikimedia.org/wiki/Deployments
>
> Compared to last year we have improved our processes considerably[2], in
> particular by making more services operate in an active/active manner,
> as well as by working on an automation and orchestration framework[3] to
> perform parallel executions across the fleet. The core of the MediaWiki
> switchover will be performed semi-automatically using a new software[4]
> that will execute all the necessary commands in sequence with little
> human involvement, and thus lowering the risk of introducing errors and
> delays.
>
> 2: https://wikitech.wikimedia.org/wiki/Switch_Datacenter
> 3: https://github.com/wikimedia/cumin
> 4: https://github.com/wikimedia/operations-switchdc
>
> Improving and automating our processes means that we're not going to be
> following the exact same steps as last year. Because of that, and
> because of other changes introduced in our environment over the course
> of the year, there is a possibility of errors creeping into the process.
> We'll certainly try to fix any issues that arise during those weeks and
> we'd like to ask everyone to be on high-alert and vigilant.
>
> To report any issues, please use one of the following channels:
>
> 1. File a Phabricator issue with project #codfw-rollout
> 2. Report issues on IRC: Freenode channel #wikimedia-tech (if urgent, or
> during the migration)
> 3. Send an e-mail to the Operations list: [hidden email] (any time)
>
> Thanks,
> Faidon
> --
> Faidon Liambotis
> Principal Operations Engineer
> Acting Director of Technical Operations
> Wikimedia Foundation
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikipedia-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikipedia-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr 17th/May 1st

Gerard Meijssen-3
Hoi,
It is great news to understand the level of our ability to switch over. I
am impressed and yes, it can and will improve but this is good to hear and
understand.
Thanks,
     GerardM

On 9 April 2017 at 03:39, zppix e <[hidden email]> wrote:

>
>
>
> > From: Faidon Liambotis <[hidden email]>
> > Date: April 7, 2017 at 8:58:09 AM CDT
> > To: Wikimedia developers <[hidden email]>
> > Subject: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr
> 17th/May 1st
> > Reply-To: Wikimedia developers <[hidden email]>
> >
> > Hi all,
> >
> > You may have heard already that, like last year, we are planning to
> > switch our active datacenter from eqiad to codfw in the week of April
> > 17th and back to eqiad two weeks later, on the week of May 1st. We do
> > this periodically in order to exercise our ability to run from the
> > backup site in case of a disaster, as well as our ability to switch
> > seamlessly to it with little user impact.
> >
> > Switching will be a gradual, multi-step process, the most visible step
> > of which will be the switch of MediaWiki application servers and
> > associated data stores. This will happen on April 19th (eqiad->codfw)
> > and May 3rd (codfw->eqiad), both at 14:00 UTC. During those windows, the
> > sites will be placed into read-only mode, for a period that we estimate
> > to last approximately 20 to 30 minutes.
> >
> > Furthermore, the deployment train will freeze for the weeks of April
> > 17th and May 1st[1], but operate normally on the week of April 24th, in
> > order to exercise our ability to deploy code while operating from the
> > backup datacenter.
> >
> > 1: https://wikitech.wikimedia.org/wiki/Deployments
> >
> > Compared to last year we have improved our processes considerably[2], in
> > particular by making more services operate in an active/active manner,
> > as well as by working on an automation and orchestration framework[3] to
> > perform parallel executions across the fleet. The core of the MediaWiki
> > switchover will be performed semi-automatically using a new software[4]
> > that will execute all the necessary commands in sequence with little
> > human involvement, and thus lowering the risk of introducing errors and
> > delays.
> >
> > 2: https://wikitech.wikimedia.org/wiki/Switch_Datacenter
> > 3: https://github.com/wikimedia/cumin
> > 4: https://github.com/wikimedia/operations-switchdc
> >
> > Improving and automating our processes means that we're not going to be
> > following the exact same steps as last year. Because of that, and
> > because of other changes introduced in our environment over the course
> > of the year, there is a possibility of errors creeping into the process.
> > We'll certainly try to fix any issues that arise during those weeks and
> > we'd like to ask everyone to be on high-alert and vigilant.
> >
> > To report any issues, please use one of the following channels:
> >
> > 1. File a Phabricator issue with project #codfw-rollout
> > 2. Report issues on IRC: Freenode channel #wikimedia-tech (if urgent, or
> > during the migration)
> > 3. Send an e-mail to the Operations list: [hidden email] (any
> time)
> >
> > Thanks,
> > Faidon
> > --
> > Faidon Liambotis
> > Principal Operations Engineer
> > Acting Director of Technical Operations
> > Wikimedia Foundation
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikipedia-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikipedia-l
>
_______________________________________________
Wikipedia-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikipedia-l
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fwd: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr 17th/May 1st

zppix e
No problem if anyone has any question feel free to email me, or ask in #wikimedia-tech on irc.freenode.net ( webchat.freenode.net )
-
Zppix
Volunteer Developer
ENWIKI rollbacker & NPR
ENWP.org/user:Zppix

Sent from my iPhone

> On Apr 8, 2017, at 11:29 PM, Gerard Meijssen <[hidden email]> wrote:
>
> Hoi,
> It is great news to understand the level of our ability to switch over. I
> am impressed and yes, it can and will improve but this is good to hear and
> understand.
> Thanks,
>     GerardM
>
>> On 9 April 2017 at 03:39, zppix e <[hidden email]> wrote:
>>
>>
>>
>>
>>> From: Faidon Liambotis <[hidden email]>
>>> Date: April 7, 2017 at 8:58:09 AM CDT
>>> To: Wikimedia developers <[hidden email]>
>>> Subject: [Wikitech-l] eqiad->codfw datacenter switchover, weeks of Apr
>> 17th/May 1st
>>> Reply-To: Wikimedia developers <[hidden email]>
>>>
>>> Hi all,
>>>
>>> You may have heard already that, like last year, we are planning to
>>> switch our active datacenter from eqiad to codfw in the week of April
>>> 17th and back to eqiad two weeks later, on the week of May 1st. We do
>>> this periodically in order to exercise our ability to run from the
>>> backup site in case of a disaster, as well as our ability to switch
>>> seamlessly to it with little user impact.
>>>
>>> Switching will be a gradual, multi-step process, the most visible step
>>> of which will be the switch of MediaWiki application servers and
>>> associated data stores. This will happen on April 19th (eqiad->codfw)
>>> and May 3rd (codfw->eqiad), both at 14:00 UTC. During those windows, the
>>> sites will be placed into read-only mode, for a period that we estimate
>>> to last approximately 20 to 30 minutes.
>>>
>>> Furthermore, the deployment train will freeze for the weeks of April
>>> 17th and May 1st[1], but operate normally on the week of April 24th, in
>>> order to exercise our ability to deploy code while operating from the
>>> backup datacenter.
>>>
>>> 1: https://wikitech.wikimedia.org/wiki/Deployments
>>>
>>> Compared to last year we have improved our processes considerably[2], in
>>> particular by making more services operate in an active/active manner,
>>> as well as by working on an automation and orchestration framework[3] to
>>> perform parallel executions across the fleet. The core of the MediaWiki
>>> switchover will be performed semi-automatically using a new software[4]
>>> that will execute all the necessary commands in sequence with little
>>> human involvement, and thus lowering the risk of introducing errors and
>>> delays.
>>>
>>> 2: https://wikitech.wikimedia.org/wiki/Switch_Datacenter
>>> 3: https://github.com/wikimedia/cumin
>>> 4: https://github.com/wikimedia/operations-switchdc
>>>
>>> Improving and automating our processes means that we're not going to be
>>> following the exact same steps as last year. Because of that, and
>>> because of other changes introduced in our environment over the course
>>> of the year, there is a possibility of errors creeping into the process.
>>> We'll certainly try to fix any issues that arise during those weeks and
>>> we'd like to ask everyone to be on high-alert and vigilant.
>>>
>>> To report any issues, please use one of the following channels:
>>>
>>> 1. File a Phabricator issue with project #codfw-rollout
>>> 2. Report issues on IRC: Freenode channel #wikimedia-tech (if urgent, or
>>> during the migration)
>>> 3. Send an e-mail to the Operations list: [hidden email] (any
>> time)
>>>
>>> Thanks,
>>> Faidon
>>> --
>>> Faidon Liambotis
>>> Principal Operations Engineer
>>> Acting Director of Technical Operations
>>> Wikimedia Foundation
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list
>>> [hidden email]
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> _______________________________________________
>> Wikipedia-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikipedia-l
>>
> _______________________________________________
> Wikipedia-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikipedia-l

_______________________________________________
Wikipedia-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikipedia-l
Loading...