Database redundancy

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Database redundancy

chris muse
We have been looking into possibly setting up a MySQL cluster to avoid some
of the complexities of a master-slave environment. Does the mediawiki code
support MySQL clustering? Has anyone tried testing a cluster with Wikipedia
or a smaller db? Also, how does wikipedia currently handle automatic
failover of a master-slave transition and then also later the slave-master
transition when the master comes back online? What do your scripts do now to
deal with that and what kind of monitoring do you to make sure everything is
in good health?

Thanks,
Chris
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Database redundancy

Brion Vibber
chris muse wrote:
> We have been looking into possibly setting up a MySQL cluster to avoid some
> of the complexities of a master-slave environment. Does the mediawiki code
> support MySQL clustering?

We don't support MySQL Cluster at this time, as it's currently limited in many
ways. (Everything must fit in memory, can't have certain field types, etc.)

> Also, how does wikipedia currently handle automatic
> failover of a master-slave transition and then also later the slave-master
> transition when the master comes back online?

We currently handle failover manually by switching things around from the
primary master to a standby. The previous master then gets cleaned up, put back
into service as a slave, and is available as a standby for the next exciting
failure.

> what kind of monitoring do you to make sure everything is
> in good health?

Not enough; servmon is currently broken and offline.

-- brion vibber (brion @ pobox.com)


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l

signature.asc (257 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Database redundancy

chris muse
> We don't support MySQL Cluster at this time, as it's currently limited in many
> ways. (Everything must fit in memory, can't have certain field types, etc.)

What mediawiki code would need to be rewritten and has anyone tried to
do a cluster with the non-supported stuff as a regular InnoDB? Is it
just the blob stuff that's a problem? I was also just kind of curious
of what the game plan was on the horizon, if you were holding out
until MySQL added support in a later version or if other things were
eclipsing database stuff at the moment. What is the general consensus
on Oracle?

Thanks,
Chris


On 3/15/06, Brion Vibber <[hidden email]> wrote:

> chris muse wrote:
> > We have been looking into possibly setting up a MySQL cluster to avoid some
> > of the complexities of a master-slave environment. Does the mediawiki code
> > support MySQL clustering?
>
> We don't support MySQL Cluster at this time, as it's currently limited in many
> ways. (Everything must fit in memory, can't have certain field types, etc.)
>
> > Also, how does wikipedia currently handle automatic
> > failover of a master-slave transition and then also later the slave-master
> > transition when the master comes back online?
>
> We currently handle failover manually by switching things around from the
> primary master to a standby. The previous master then gets cleaned up, put back
> into service as a slave, and is available as a standby for the next exciting
> failure.
>
> > what kind of monitoring do you to make sure everything is
> > in good health?
>
> Not enough; servmon is currently broken and offline.
>
> -- brion vibber (brion @ pobox.com)
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> http://mail.wikipedia.org/mailman/listinfo/wikitech-l
>
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Database redundancy

Brion Vibber
chris muse wrote:
> What mediawiki code would need to be rewritten and has anyone tried to
> do a cluster with the non-supported stuff as a regular InnoDB? Is it
> just the blob stuff that's a problem? I was also just kind of curious
> of what the game plan was on the horizon, if you were holding out
> until MySQL added support in a later version or if other things were
> eclipsing database stuff at the moment.

Currently waiting for the upcoming versions which allow disk-backed data, etc.
No point in rushing in when our own contributors who are MySQL employees are
telling us to wait for it to mature a bit. :)

> What is the general consensus on Oracle?

From what I hear on the interweb, they are devil-people who eat babies.

-- brion vibber (brion @ pobox.com)


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l

signature.asc (257 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Database redundancy

Jay Ashworth-2
On Wed, Mar 15, 2006 at 01:00:26PM -0800, Brion Vibber wrote:
> > What is the general consensus on Oracle?
>
> From what I hear on the interweb, they are devil-people who eat babies.

I believe you mispelt "intarweb".

(And "teh", come to think of it)

How about PostGreSQL?

Cheers,
-- jra
--
Jay R. Ashworth                                                [hidden email]
Designer                          Baylink                             RFC 2100
Ashworth & Associates        The Things I Think                        '87 e24
St Petersburg FL USA      http://baylink.pitas.com             +1 727 647 1274

        A: No.
        Q: Should I include quotations after my message body?
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Database redundancy

Gregory Maxwell
On 3/16/06, Jay R. Ashworth <[hidden email]> wrote:
> How about PostGreSQL?

Working on porting Wikipedia to PostgreSQL is a good way to get hired
working for MySQL AB. Seems so far the only group thats been immune to
it are the folks at http://pgfoundry.org/projects/wikipedia

As far as your question goes, PG is still stuck with master/slave
replication as well..  True multimaster was accomplished in Postres-R
as a proof of concept, but hasn't been synced up and made part of the
official code yet. Someday.

For the queryload that Wikipedia creates, MySQL runs fine... although,
for adhoc queries PG wipes the floor (no bitmap joins in mysql
@#$@#$). Also, if you import Wikipedia into PG it's much smaller than
MySQL even with gzip disabled because PG gzips on the back end and
doesn't insert airspace in the tables (if you want your tables
physically clustered on some field, you have to configure that
explicitly, it's nor forced like w/ inno and primary keys). Also,
since PG has proper UTF-8 support you don't need to use blobs... thus
it's easy to throw in full text indexes and perform (index
accelerated) queries on article text.

Related to this I've mostly given up on using toolserver for analysis
stuff... between the current unavailability of article text, the
unresponsiveness of the folks operating it, and the poor performance
of MySQL with complex queries... .. Being forced to choose between
working with old data that I can query flexibly and painlessly and
working with fresh data that is painful to query.. :(
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Database redundancy

Robert Treat
In reply to this post by Jay Ashworth-2
On Thursday 16 March 2006 11:57, Jay R. Ashworth wrote:
> On Wed, Mar 15, 2006 at 01:00:26PM -0800, Brion Vibber wrote:
> > > What is the general consensus on Oracle?
> >
> > From what I hear on the interweb, they are devil-people who eat babies.
>

:-)

> I believe you mispelt "intarweb".
>
> (And "teh", come to think of it)
>
> How about PostGreSQL?
>

We've seen the PostgreSQL port tested with Slony replication in a simple
master->slave setup and it worked well on the surface (This was with the
wikipedia dump data, but I believe without binary data) but I don't want to
claim it was exhaustive or anything.  

--
Robert Treat
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
_______________________________________________
Wikitech-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/wikitech-l