DEPRECATION WARNING: dbstore1002 is going to be decommissioned on March 4th

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

DEPRECATION WARNING: dbstore1002 is going to be decommissioned on March 4th

Luca Toscano
Hi everybody,

the Analytics team has been working with the SRE Data Persistence team
during the last months to replace dbstore1002 with three brand new nodes,
dbstore100[3-5]. We are moving from a single mysql instance (multi-source)
to a multi-instance environment.

For more info please check:
* T210478 and related subtasks.
* https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas

We are planning to decommission the dbstore1002 host (namely stopping mysql
and shutting down the server) on Monday March 4th (EU morning). We have
recently been following up with a lot of users to help them migrate to the
new environment, so we are reasonably sure that this move should not
heavily impact anybody, but if we have left some use case aside please let
us know in https://phabricator.wikimedia.org/T215589. If we don't hear
anything before the March 4th deadline we'll proceed with the host
decommission maintenance.

Luca (on behalf of the Analytics team)
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: [Analytics] DEPRECATION WARNING: dbstore1002 is going to be decommissioned on March 4th

Leila Zia
On Fri, Feb 22, 2019 at 2:45 AM Luca Toscano <[hidden email]> wrote:
>
> the Analytics team has been working with the SRE Data Persistence team during the last months to replace dbstore1002 with three brand new nodes, dbstore100[3-5]. We are moving from a single mysql instance (multi-source) to a multi-instance environment.

This has been an incredible amount of work, both in socializing the
idea and also implementation and making sure workflows don't break as
much as possible. Thank you to all of you who worked on this over the
past months and to those of you who maintained the single mysql
instance for all the past years. I hope that the maintenance workflows
become easier for those of you who continue to maintain these systems
for us.

Thank you! :)

Best,
Leila

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: [Analytics] DEPRECATION WARNING: dbstore1002 is going to be decommissioned on March 4th

Aaron Halfaker-3
+1 to everything Leila said.  This is very difficult work and I'm amazed to
see it happen so smoothly -- at least from my point of view :)

On Fri, Feb 22, 2019 at 10:07 AM Leila Zia <[hidden email]> wrote:

> On Fri, Feb 22, 2019 at 2:45 AM Luca Toscano <[hidden email]>
> wrote:
> >
> > the Analytics team has been working with the SRE Data Persistence team
> during the last months to replace dbstore1002 with three brand new nodes,
> dbstore100[3-5]. We are moving from a single mysql instance (multi-source)
> to a multi-instance environment.
>
> This has been an incredible amount of work, both in socializing the
> idea and also implementation and making sure workflows don't break as
> much as possible. Thank you to all of you who worked on this over the
> past months and to those of you who maintained the single mysql
> instance for all the past years. I hope that the maintenance workflows
> become easier for those of you who continue to maintain these systems
> for us.
>
> Thank you! :)
>
> Best,
> Leila
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>


--

Aaron Halfaker

Principal Research Scientist

Head of the Scoring Platform team
Wikimedia Foundation
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Reply | Threaded
Open this post in threaded view
|

Re: [Analytics] DEPRECATION WARNING: dbstore1002 is going to be decommissioned on March 4th

Neil Patel Quinn
Let me join in with a late appreciation of all the work that Analytics has
done making this migration happen.

As probably the heaviest users of the MediaWiki replicas, we in the Product
Analytics team were a bit worried when we heard the initial plan and had
lot of requests for how to make the new infrastructure usable. Analytics
listened to us in meeting after meeting and in long Phab conversations and
worked *hard *to come up with creative, feasible solutions for our
challenges, even as dbstore1002 slowly failed under them.

In particular, Luca did amazing work during this process: tracking all five
of us analysts down during the mayhem of all hands to see how we worked
with dbstore1002, coming up with smart hacks with SRV records and
command-line clients to make it easier for us to work across the multiple
replacement hosts, exhaustively documenting the new setup
<https://wikitech.wikimedia.org/w/index.php?title=Analytics/Data_access#MariaDB_replicas>,
and more.

Thank you so much to all of you for providing the infrastructure that
literally makes our jobs possible! 🎉

On Fri, 22 Feb 2019 at 21:52, Aaron Halfaker <[hidden email]>
wrote:

> +1 to everything Leila said.  This is very difficult work and I'm amazed
> to see it happen so smoothly -- at least from my point of view :)
>
> On Fri, Feb 22, 2019 at 10:07 AM Leila Zia <[hidden email]> wrote:
>
>> On Fri, Feb 22, 2019 at 2:45 AM Luca Toscano <[hidden email]>
>> wrote:
>> >
>> > the Analytics team has been working with the SRE Data Persistence team
>> during the last months to replace dbstore1002 with three brand new nodes,
>> dbstore100[3-5]. We are moving from a single mysql instance (multi-source)
>> to a multi-instance environment.
>>
>> This has been an incredible amount of work, both in socializing the
>> idea and also implementation and making sure workflows don't break as
>> much as possible. Thank you to all of you who worked on this over the
>> past months and to those of you who maintained the single mysql
>> instance for all the past years. I hope that the maintenance workflows
>> become easier for those of you who continue to maintain these systems
>> for us.
>>
>> Thank you! :)
>>
>> Best,
>> Leila
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
>
> --
>
> Aaron Halfaker
>
> Principal Research Scientist
>
> Head of the Scoring Platform team
> Wikimedia Foundation
>
_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l