Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

CT Woo
All,

The switchover work is done.

The site was was available to readers throughout the migration work though
it was in read-only mode for about 32 minutes, when Asher and Mark had to
migrate the database masters over from Tampa to Ashburn.

We will cancel the reminding maintenance windows.

Thank you all for your patience and understanding.

Regards,
CT Woo

On Sat, Jan 19, 2013 at 10:57 AM, Ct Woo <[hidden email]> wrote:

>  All,
>
> We will be proceeding with the datacenter switchover plan this coming
> Tuesday (Jan 22, 2013), unless we discover some unexpected and
> insurmountable issues in our tests between now and then.
>
> During the 8-hour migration window on the 22nd, 23rd and 24th (from 17:00
> UTC to 01:00 UTC hours  / 9am to 5pm PST),  there would be times (lasting
> about 30 minutes) where the site would be set to "read-only" mode, to
> facilitate master database switchovers from one datacenter to another.
> While the site should be available to readers, no new contents could be
> created, edited or uploaded.
>
> We are aware of the inconvenience and we have put together plans to
> minimize such annoyances, e.g., automating much of the procedures,
> mitigating known risks,  and performing tests to identify issues prior to
> deployment. Given the scale and complexity of this migration, we do realize
> not all operational impact is predictable.  Some users could experience
> intermittent site unavailability and/or performance issues unfortunately.
>
> You can follow the migration on chat.freenode.net<http://irc.freenode.net>
> (and not irc.freenode.org as mentioned in previous email) in the
> #wikimedia-operations channel.
>
> Thanks,
> CT Woo
>
> ---------- Forwarded message ----------
> From: Ct Woo <[hidden email]>
> Date: Fri, Jan 11, 2013 at 12:07 PM
> Subject: Update on Ashburn data center switchover / migration – target
> date is week of 1/22/13
> To: Wikimedia developers <[hidden email]>, Development
> and Operations Engineers <[hidden email]>
>
>
> All,
>
> The Migration team is in the last lap on completing the remaining tasks to
> ready our software stack and Ashburn infrastructure for the big
> switchover day.
>
> Per my last update,<http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063668.html>
> with the Fundraising activity behind us now, the team has scheduled the *week
> of 22nd January*, 2013 to perform the switchover. We are going to block a
> 8-hour migration window on the *22nd, 23rd and 24**th*.  During those
> periods, *17:00 UTC to 01:00 UTC hours (9am to 5pm PST*), there will be
> intermittent blackouts and they will be treated as 'planned' outages.  You
> can follow the migration on irc.freenode.org in the #wikimedia-operations
> channel.
>
> The team is putting the finishing touches to the last few tasks and we
> will make the final Go/No decision on 18th Jan, 2013. An update will send
> out then. For those interested in tracking the progress, the meeting notes
> are captured on this wikitech page<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning#Improving_Switchover>
> .
>
> *Please note that we will be restricting code deployment during that
> week, allowing only emergency and critical ones only.*
>
> Thanks.
>
> CT Woo
>
>
>
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Brion Vibber
On Tue, Jan 22, 2013 at 2:53 PM, Ct Woo <[hidden email]> wrote:

> The switchover work is done.
>
> The site was was available to readers throughout the migration work though
> it was in read-only mode for about 32 minutes, when Asher and Mark had to
> migrate the database masters over from Tampa to Ashburn.
>
> We will cancel the reminding maintenance windows.
>

Note that test.wikipedia.org is still in read-only mode <
https://bugzilla.wikimedia.org/show_bug.cgi?id=44259>.

Folks needing a read/write test wiki may use test2.wikipedia.org for now,
which runs on a different configuration.

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Brian Wolff
In reply to this post by CT Woo
Good work to everyone involved!

-bawolff
On 2013-01-22 6:53 PM, "Ct Woo" <[hidden email]> wrote:

> All,
>
> The switchover work is done.
>
> The site was was available to readers throughout the migration work though
> it was in read-only mode for about 32 minutes, when Asher and Mark had to
> migrate the database masters over from Tampa to Ashburn.
>
> We will cancel the reminding maintenance windows.
>
> Thank you all for your patience and understanding.
>
> Regards,
> CT Woo
>
> On Sat, Jan 19, 2013 at 10:57 AM, Ct Woo <[hidden email]> wrote:
>
> >  All,
> >
> > We will be proceeding with the datacenter switchover plan this coming
> > Tuesday (Jan 22, 2013), unless we discover some unexpected and
> > insurmountable issues in our tests between now and then.
> >
> > During the 8-hour migration window on the 22nd, 23rd and 24th (from 17:00
> > UTC to 01:00 UTC hours  / 9am to 5pm PST),  there would be times (lasting
> > about 30 minutes) where the site would be set to "read-only" mode, to
> > facilitate master database switchovers from one datacenter to another.
> > While the site should be available to readers, no new contents could be
> > created, edited or uploaded.
> >
> > We are aware of the inconvenience and we have put together plans to
> > minimize such annoyances, e.g., automating much of the procedures,
> > mitigating known risks,  and performing tests to identify issues prior to
> > deployment. Given the scale and complexity of this migration, we do
> realize
> > not all operational impact is predictable.  Some users could experience
> > intermittent site unavailability and/or performance issues unfortunately.
> >
> > You can follow the migration on chat.freenode.net<
> http://irc.freenode.net>
> > (and not irc.freenode.org as mentioned in previous email) in the
> > #wikimedia-operations channel.
> >
> > Thanks,
> > CT Woo
> >
> > ---------- Forwarded message ----------
> > From: Ct Woo <[hidden email]>
> > Date: Fri, Jan 11, 2013 at 12:07 PM
> > Subject: Update on Ashburn data center switchover / migration – target
> > date is week of 1/22/13
> > To: Wikimedia developers <[hidden email]>, Development
> > and Operations Engineers <[hidden email]>
> >
> >
> > All,
> >
> > The Migration team is in the last lap on completing the remaining tasks
> to
> > ready our software stack and Ashburn infrastructure for the big
> > switchover day.
> >
> > Per my last update,<
> http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063668.html>
> > with the Fundraising activity behind us now, the team has scheduled the
> *week
> > of 22nd January*, 2013 to perform the switchover. We are going to block a
> > 8-hour migration window on the *22nd, 23rd and 24**th*.  During those
> > periods, *17:00 UTC to 01:00 UTC hours (9am to 5pm PST*), there will be
> > intermittent blackouts and they will be treated as 'planned' outages.
>  You
> > can follow the migration on irc.freenode.org in the
> #wikimedia-operations
> > channel.
> >
> > The team is putting the finishing touches to the last few tasks and we
> > will make the final Go/No decision on 18th Jan, 2013. An update will send
> > out then. For those interested in tracking the progress, the meeting
> notes
> > are captured on this wikitech page<
> http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning#Improving_Switchover
> >
> > .
> >
> > *Please note that we will be restricting code deployment during that
> > week, allowing only emergency and critical ones only.*
> >
> > Thanks.
> >
> > CT Woo
> >
> >
> >
> >
> >
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Arthur Richards-2
In reply to this post by CT Woo
Wow, awesome, congratulations all!

On Tue, Jan 22, 2013 at 3:53 PM, Ct Woo <[hidden email]> wrote:

> We will cancel the reminding maintenance windows.
>

Does this mean the deployment freeze for the rest of the week has been
thawed, or are deployments still on hold until next week?

--
Arthur Richards
Software Engineer, Mobile
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Ori Livneh
In reply to this post by Brian Wolff
Congrats, guys and gals.  

--
Ori Livneh


On Tuesday, January 22, 2013 at 3:22 PM, Bawolff Bawolff wrote:

> Good work to everyone involved!
>  
> -bawolff
> On 2013-01-22 6:53 PM, "Ct Woo" <[hidden email] (mailto:[hidden email])> wrote:
>  
> > All,
> >  
> > The switchover work is done.
> >  
> > The site was was available to readers throughout the migration work though
> > it was in read-only mode for about 32 minutes, when Asher and Mark had to
> > migrate the database masters over from Tampa to Ashburn.
> >  
> > We will cancel the reminding maintenance windows.
> >  
> > Thank you all for your patience and understanding.
> >  
> > Regards,
> > CT Woo
> >  
> > On Sat, Jan 19, 2013 at 10:57 AM, Ct Woo <[hidden email] (mailto:[hidden email])> wrote:
> >  
> > > All,
> > >  
> > > We will be proceeding with the datacenter switchover plan this coming
> > > Tuesday (Jan 22, 2013), unless we discover some unexpected and
> > > insurmountable issues in our tests between now and then.
> > >  
> > > During the 8-hour migration window on the 22nd, 23rd and 24th (from 17:00
> > > UTC to 01:00 UTC hours / 9am to 5pm PST), there would be times (lasting
> > > about 30 minutes) where the site would be set to "read-only" mode, to
> > > facilitate master database switchovers from one datacenter to another.
> > > While the site should be available to readers, no new contents could be
> > > created, edited or uploaded.
> > >  
> > > We are aware of the inconvenience and we have put together plans to
> > > minimize such annoyances, e.g., automating much of the procedures,
> > > mitigating known risks, and performing tests to identify issues prior to
> > > deployment. Given the scale and complexity of this migration, we do
> >  
> >  
> > realize
> > > not all operational impact is predictable. Some users could experience
> > > intermittent site unavailability and/or performance issues unfortunately.
> > >  
> > > You can follow the migration on chat.freenode.net (http://chat.freenode.net)<
> > http://irc.freenode.net>
> > > (and not irc.freenode.org (http://irc.freenode.org) as mentioned in previous email) in the
> > > #wikimedia-operations channel.
> > >  
> > > Thanks,
> > > CT Woo
> > >  
> > > ---------- Forwarded message ----------
> > > From: Ct Woo <[hidden email] (mailto:[hidden email])>
> > > Date: Fri, Jan 11, 2013 at 12:07 PM
> > > Subject: Update on Ashburn data center switchover / migration – target
> > > date is week of 1/22/13
> > > To: Wikimedia developers <[hidden email] (mailto:[hidden email])>, Development
> > > and Operations Engineers <[hidden email] (mailto:[hidden email])>
> > >  
> > >  
> > > All,
> > >  
> > > The Migration team is in the last lap on completing the remaining tasks
> > to
> > > ready our software stack and Ashburn infrastructure for the big
> > > switchover day.
> > >  
> > > Per my last update,<
> > http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063668.html>
> > > with the Fundraising activity behind us now, the team has scheduled the
> >  
> >  
> > *week
> > > of 22nd January*, 2013 to perform the switchover. We are going to block a
> > > 8-hour migration window on the *22nd, 23rd and 24**th*. During those
> > > periods, *17:00 UTC to 01:00 UTC hours (9am to 5pm PST*), there will be
> > > intermittent blackouts and they will be treated as 'planned' outages.
> >  
> >  
> > You
> > > can follow the migration on irc.freenode.org (http://irc.freenode.org) in the
> >  
> >  
> > #wikimedia-operations
> > > channel.
> > >  
> > > The team is putting the finishing touches to the last few tasks and we
> > > will make the final Go/No decision on 18th Jan, 2013. An update will send
> > > out then. For those interested in tracking the progress, the meeting
> >  
> >  
> > notes
> > > are captured on this wikitech page<
> >  
> >  
> > http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning#Improving_Switchover
> > >  
> > > .
> > >  
> > > *Please note that we will be restricting code deployment during that
> > > week, allowing only emergency and critical ones only.*
> > >  
> > > Thanks.
> > >  
> > > CT Woo
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email] (mailto:[hidden email])
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>  
>  
> _______________________________________________
> Wikitech-l mailing list
> [hidden email] (mailto:[hidden email])
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Tomasz Finc-2
In reply to this post by CT Woo
So seamless. Well done!

--tomasz


On Tue, Jan 22, 2013 at 2:53 PM, Ct Woo <[hidden email]> wrote:

> All,
>
> The switchover work is done.
>
> The site was was available to readers throughout the migration work though
> it was in read-only mode for about 32 minutes, when Asher and Mark had to
> migrate the database masters over from Tampa to Ashburn.
>
> We will cancel the reminding maintenance windows.
>
> Thank you all for your patience and understanding.
>
> Regards,
> CT Woo
>
> On Sat, Jan 19, 2013 at 10:57 AM, Ct Woo <[hidden email]> wrote:
>
>>  All,
>>
>> We will be proceeding with the datacenter switchover plan this coming
>> Tuesday (Jan 22, 2013), unless we discover some unexpected and
>> insurmountable issues in our tests between now and then.
>>
>> During the 8-hour migration window on the 22nd, 23rd and 24th (from 17:00
>> UTC to 01:00 UTC hours  / 9am to 5pm PST),  there would be times (lasting
>> about 30 minutes) where the site would be set to "read-only" mode, to
>> facilitate master database switchovers from one datacenter to another.
>> While the site should be available to readers, no new contents could be
>> created, edited or uploaded.
>>
>> We are aware of the inconvenience and we have put together plans to
>> minimize such annoyances, e.g., automating much of the procedures,
>> mitigating known risks,  and performing tests to identify issues prior to
>> deployment. Given the scale and complexity of this migration, we do realize
>> not all operational impact is predictable.  Some users could experience
>> intermittent site unavailability and/or performance issues unfortunately.
>>
>> You can follow the migration on chat.freenode.net<http://irc.freenode.net>
>> (and not irc.freenode.org as mentioned in previous email) in the
>> #wikimedia-operations channel.
>>
>> Thanks,
>> CT Woo
>>
>> ---------- Forwarded message ----------
>> From: Ct Woo <[hidden email]>
>> Date: Fri, Jan 11, 2013 at 12:07 PM
>> Subject: Update on Ashburn data center switchover / migration – target
>> date is week of 1/22/13
>> To: Wikimedia developers <[hidden email]>, Development
>> and Operations Engineers <[hidden email]>
>>
>>
>> All,
>>
>> The Migration team is in the last lap on completing the remaining tasks
>> to ready our software stack and Ashburn infrastructure for the big
>> switchover day.
>>
>> Per my last update,<http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063668.html>
>> with the Fundraising activity behind us now, the team has scheduled the *week
>> of 22nd January*, 2013 to perform the switchover. We are going to block
>> a 8-hour migration window on the *22nd, 23rd and 24**th*.  During those
>> periods, *17:00 UTC to 01:00 UTC hours (9am to 5pm PST*), there will be
>> intermittent blackouts and they will be treated as 'planned' outages.  You
>> can follow the migration on irc.freenode.org in the
>> #wikimedia-operations channel.
>>
>> The team is putting the finishing touches to the last few tasks and we
>> will make the final Go/No decision on 18th Jan, 2013. An update will
>> send out then. For those interested in tracking the progress, the meeting
>> notes are captured on this wikitech page<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning#Improving_Switchover>
>> .
>>
>> *Please note that we will be restricting code deployment during that
>> week, allowing only emergency and critical ones only.*
>>
>> Thanks.
>>
>> CT Woo
>>
>>
>>
>>
>>
>
> _______________________________________________
> Engineering mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Risker
Well done, very pleased to see that the careful planning paid off so well.
I raise a glass in honour of those who worked so hard to make this an
almost invisible operation.

Risker

On 22 January 2013 19:04, Tomasz Finc <[hidden email]> wrote:

> So seamless. Well done!
>
> --tomasz
>
>
> On Tue, Jan 22, 2013 at 2:53 PM, Ct Woo <[hidden email]> wrote:
>
> > All,
> >
> > The switchover work is done.
> >
> > The site was was available to readers throughout the migration work
> though
> > it was in read-only mode for about 32 minutes, when Asher and Mark had to
> > migrate the database masters over from Tampa to Ashburn.
> >
> > We will cancel the reminding maintenance windows.
> >
> > Thank you all for your patience and understanding.
> >
> > Regards,
> > CT Woo
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week of 1/22/13

Tim Starling-2
In reply to this post by Brion Vibber
On 23/01/13 10:05, Brion Vibber wrote:
> Note that test.wikipedia.org is still in read-only mode <
> https://bugzilla.wikimedia.org/show_bug.cgi?id=44259>.

Fixed.

-- Tim Starling



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l