Toolserver limitation to come?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Toolserver limitation to come?

Tim Landscheidt
Hi,

at the office hour yesterday
(cf. http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt):

| [...]
| <Coren> multichill: The long story short; replicating
|         databases is happening soon (Within the month)
|         Replicating multiple copies of commons and wikidata
|         isn't going to happen that way; it needs to be built
|         into application logic or using federated tables.
|         Almost /all/ of the problems the TS have had with
|         replication were caused by that redundancy and
|         trying to keep it synced.
| <multichill> Coren: So you're basically saying Toollabs is
|              useless for me
| <Platonides> {{ref needed}}
| <Coren> We're all more than happy to help you (and any other
|         maintainer) with adapting your tools to work in that
|         setup.
| <scfc_de> Coren: In that case, Tools will not be able to
|           replace Toolserver.
| <giftpfla1ze> what are federated tables btw?
| <Ryan_Lane> AFAIK toolserver will also have this limitation
|             at some point
| <scfc_de> Ryan_Lane: What do you mean?
| [...]

There were never answers to this, so I bring it up here
again:

1. How were "almost /all/ of the problems the TS have had
   with replication" "caused by that redundancy and trying
   to keep it synced"?

2. What limitation will the Toolserver have at some point?

Tim


_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Toolserver limitation to come?

Ryan Lane-3
On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt <[hidden email]> wrote:
Hi,

at the office hour yesterday
(cf. http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt):

| [...]
| <Coren> multichill: The long story short; replicating
|         databases is happening soon (Within the month)
|         Replicating multiple copies of commons and wikidata
|         isn't going to happen that way; it needs to be built
|         into application logic or using federated tables.
|         Almost /all/ of the problems the TS have had with
|         replication were caused by that redundancy and
|         trying to keep it synced.
| <multichill> Coren: So you're basically saying Toollabs is
|              useless for me
| <Platonides> {{ref needed}}
| <Coren> We're all more than happy to help you (and any other
|         maintainer) with adapting your tools to work in that
|         setup.
| <scfc_de> Coren: In that case, Tools will not be able to
|           replace Toolserver.
| <giftpfla1ze> what are federated tables btw?
| <Ryan_Lane> AFAIK toolserver will also have this limitation
|             at some point
| <scfc_de> Ryan_Lane: What do you mean?
| [...]

There were never answers to this, so I bring it up here
again:

1. How were "almost /all/ of the problems the TS have had
   with replication" "caused by that redundancy and trying
   to keep it synced"?

2. What limitation will the Toolserver have at some point?


As to #2: From what I've been told this has to do with future sharding plans for the databases, and due to a change in how we'll be doing replication. Of course, I've heard this in passing. For answers to both of these questions you'll need to talk to binasher and/or notpeter on IRC, as they are the ones doing the database work.

- Ryan

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Toolserver limitation to come?

Patricia Pintilie

<a rel="license" href="http://creativecommons.org/licenses/by/3.0/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by/3.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/3.0/">Creative Commons Attribution 3.0 Unported License</a>.

On May 1, 2013 8:49 PM, "Ryan Lane" <[hidden email]> wrote:
On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt <[hidden email]> wrote:
Hi,

at the office hour yesterday
(cf. http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt):

| [...]
| <Coren> multichill: The long story short; replicating
|         databases is happening soon (Within the month)
|         Replicating multiple copies of commons and wikidata
|         isn't going to happen that way; it needs to be built
|         into application logic or using federated tables.
|         Almost /all/ of the problems the TS have had with
|         replication were caused by that redundancy and
|         trying to keep it synced.
| <multichill> Coren: So you're basically saying Toollabs is
|              useless for me
| <Platonides> {{ref needed}}
| <Coren> We're all more than happy to help you (and any other
|         maintainer) with adapting your tools to work in that
|         setup.
| <scfc_de> Coren: In that case, Tools will not be able to
|           replace Toolserver.
| <giftpfla1ze> what are federated tables btw?
| <Ryan_Lane> AFAIK toolserver will also have this limitation
|             at some point
| <scfc_de> Ryan_Lane: What do you mean?
| [...]

There were never answers to this, so I bring it up here
again:

1. How were "almost /all/ of the problems the TS have had
   with replication" "caused by that redundancy and trying
   to keep it synced"?

2. What limitation will the Toolserver have at some point?


As to #2: From what I've been told this has to do with future sharding plans for the databases, and due to a change in how we'll be doing replication. Of course, I've heard this in passing. For answers to both of these questions you'll need to talk to binasher and/or notpeter on IRC, as they are the ones doing the database work.

- Ryan

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Toolserver limitation to come?

Patricia Pintilie

Remove wiki offline
Do not allow diffrent languages on same feeds
Delete unused accounts
Delete dangerous information
Acknowledge that the sole creator could be anyone, and expects more order in a hectic database with too much junk being reedited over and over.
Once its written correctly do not allow repeaters to do over incorrectly work that was ligit.
Figure out how many angles are approaching the database.
This toolserver can cause havoc or can contain order depends on proper knowledge used in operating it.
Thanks
MILASTAR.TS.RO

On May 1, 2013 9:17 PM, "Patricia Pintilie" <[hidden email]> wrote:

<a rel="license" href="http://creativecommons.org/licenses/by/3.0/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by/3.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/3.0/">Creative Commons Attribution 3.0 Unported License</a>.

On May 1, 2013 8:49 PM, "Ryan Lane" <[hidden email]> wrote:
On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt <[hidden email]> wrote:
Hi,

at the office hour yesterday
(cf. http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt):

| [...]
| <Coren> multichill: The long story short; replicating
|         databases is happening soon (Within the month)
|         Replicating multiple copies of commons and wikidata
|         isn't going to happen that way; it needs to be built
|         into application logic or using federated tables.
|         Almost /all/ of the problems the TS have had with
|         replication were caused by that redundancy and
|         trying to keep it synced.
| <multichill> Coren: So you're basically saying Toollabs is
|              useless for me
| <Platonides> {{ref needed}}
| <Coren> We're all more than happy to help you (and any other
|         maintainer) with adapting your tools to work in that
|         setup.
| <scfc_de> Coren: In that case, Tools will not be able to
|           replace Toolserver.
| <giftpfla1ze> what are federated tables btw?
| <Ryan_Lane> AFAIK toolserver will also have this limitation
|             at some point
| <scfc_de> Ryan_Lane: What do you mean?
| [...]

There were never answers to this, so I bring it up here
again:

1. How were "almost /all/ of the problems the TS have had
   with replication" "caused by that redundancy and trying
   to keep it synced"?

2. What limitation will the Toolserver have at some point?


As to #2: From what I've been told this has to do with future sharding plans for the databases, and due to a change in how we'll be doing replication. Of course, I've heard this in passing. For answers to both of these questions you'll need to talk to binasher and/or notpeter on IRC, as they are the ones doing the database work.

- Ryan

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Toolserver limitation to come?

Marlen Caemmerer-3
In reply to this post by Ryan Lane-3
Hey,


On Wed, 1 May 2013, Ryan Lane wrote:

>
> On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt <[hidden email]>wrote:
>
>> | [...]
>>
>> There were never answers to this, so I bring it up here
>> again:
>>
>> 1. How were "almost /all/ of the problems the TS have had
>>    with replication" "caused by that redundancy and trying
>>    to keep it synced"?


Thanks for this question :) - I also want to know.
From my perspective it does not look like this and even the data inconsistencies appear when we have no commons copy on a mysql instance.
And: DaB experimented with federated tables for commons too and we decided to not do this since it does not perform from the start.
Probably nowadays when I planned something new in this area (which does not seem to make sense for TS) I'd really give Galera a try -
http://codership.com/content/using-galera-cluster

>>
>> 2. What limitation will the Toolserver have at some point?
>>
>>
> As to #2: From what I've been told this has to do with future sharding
> plans for the databases, and due to a change in how we'll be doing
> replication. Of course, I've heard this in passing. For answers to both of
> these questions you'll need to talk to binasher and/or notpeter on IRC, as
> they are the ones doing the database work.


Thanks for telling...

Cheers
  Marlen/nosy

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Toolserver limitation to come?

Platonides
On 02/05/13 10:03, Marlen Caemmerer wrote:

> Hey,
>
> On Wed, 1 May 2013, Ryan Lane wrote:
>> On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt
>> <[hidden email]>wrote:
>>>
>>> There were never answers to this, so I bring it up here
>>> again:
>>>
>>> 1. How were "almost /all/ of the problems the TS have had
>>>    with replication" "caused by that redundancy and trying
>>>    to keep it synced"?
>
> Thanks for this question :) - I also want to know.
> From my perspective it does not look like this and even the data
> inconsistencies appear when we have no commons copy on a mysql instance.
> And: DaB experimented with federated tables for commons too and we
> decided to not do this since it does not perform from the start.
> Probably nowadays when I planned something new in this area (which does
> not seem to make sense for TS) I'd really give Galera a try -
> http://codership.com/content/using-galera-cluster

FWIW, my {{ref needed}} phrase in the log was also intended to that
statement by Coren, not to multichill reply.

Additionally, I rembember you data inconsistencies happen even with
native mysql replication (as informed by Nosy earlier).

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Toolserver limitation to come?

Patricia Pintilie
In reply to this post by Marlen Caemmerer-3


On May 2, 2013 3:08 AM, "Marlen Caemmerer" <[hidden email]> wrote:
>
> Hey,
>
>
>
> On Wed, 1 May 2013, Ryan Lane wrote:
>
>>
>> On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt <[hidden email]>wrote:
>>
>>> | [...]
>>>
>>> There were never answers to this, so I bring it up here
>>> again:
>>>
>>> 1. How were "almost /all/ of the problems the TS have had
>>>    with replication" "caused by that redundancy and trying
>>>    to keep it synced"?
>
>Ryan, repeaters are from the root of a program inwhich start the initial setup.
>
> Thanks for this question :) - I also want to know.
> From my perspective it does not look like this and even the data inconsistencies appear when we have no commons copy on a mysql instance.
> And: DaB experimented with federated tables for commons too and we decided to not do this since it does not perform from the start.
> Probably nowadays when I planned something new in this area (which does not seem to make sense for TS) I'd really give Galera a try - http://codership.com/content/using-galera-cluster
>
>
>>>
>>> 2. What limitation will the Toolserver have at some point?
>>>
>>>
>> As to #2: From what I've been told this has to do with future sharding
>> plans for the databases, and due to a change in how we'll be doing
>> replication. Of course, I've heard this in passing. For answers to both of
>> these questions you'll need to talk to binasher and/or notpeter on IRC, as
>> they are the ones doing the database work.
>
>
>
> Thanks for telling...
>
> Cheers
>         Marlen/nosy
>
>
> _______________________________________________
> Toolserver-l mailing list ([hidden email])
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette


_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette