Better non-MySQL db support

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Better non-MySQL db support

Mark A. Hershberger-2
How can we improve the support for databases like PostgreSQL, Oracle,
DB2 and MS SQL?

Getting Jenkins involved in testing isn't the (only) answer, though it
would certainly help.

If developers who were interested in those databases could watch
includes/db, that would help, as well.

If nothing else, I will make an effort to get the developers for those
databases involved in RC testing and make the RC available a month
before release.

--
http://hexmode.com/

There is no path to peace. Peace is the path.
   -- Mahatma Gandhi, "Non-Violence in Peace and War"

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Petr Bena
I would be happy to install some mediawiki on oracle db, but I have no
oracle db on any of my personal servers :/

The main problem of oracle is that it's not very much free - thus it's
not packaged by most of linux vendors and it might be hard to install
for many sysadmins. (In order to install oracle, you don't just need a
system that meets all the technical parameters, but also good
understanding of response files used by their universal installer).

From my experience, installing oracle db on debian or ubuntu is pretty
complicated (installation on oracle linux and such is not that hard
though - it comes with exact version of packages that oracle
requires).


However - it's not a freeware and that automatically makes it very
unpopular for open source OS vendors and very unlikely a choice of
standard webadmin and that makes it hard to test. Even if I was able
to obtain some license to install oracle on any of my servers, I
wouldn't do it as it eats too much resources. On other hand if oracle
granted some license to wikimedia, we could set up a project on
wikimedia labs for this purpose.

On Mon, Feb 25, 2013 at 4:27 PM, Mark A. Hershberger <[hidden email]> wrote:

> How can we improve the support for databases like PostgreSQL, Oracle,
> DB2 and MS SQL?
>
> Getting Jenkins involved in testing isn't the (only) answer, though it
> would certainly help.
>
> If developers who were interested in those databases could watch
> includes/db, that would help, as well.
>
> If nothing else, I will make an effort to get the developers for those
> databases involved in RC testing and make the RC available a month
> before release.
>
> --
> http://hexmode.com/
>
> There is no path to peace. Peace is the path.
>    -- Mahatma Gandhi, "Non-Violence in Peace and War"
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
OQ
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

OQ
In reply to this post by Mark A. Hershberger-2
On Mon, Feb 25, 2013 at 9:27 AM, Mark A. Hershberger <[hidden email]> wrote:
> How can we improve the support for databases like PostgreSQL, Oracle,
> DB2 and MS SQL?
>

The main issues arise from not keeping these other DBs in mind when
writing the queries (and also not using the db layer).

Things like quoting timestamps, more strict adherence to the SQL
spec(s), stricter data type checking, etc.

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Chad
In reply to this post by Mark A. Hershberger-2
On Mon, Feb 25, 2013 at 7:27 AM, Mark A. Hershberger <[hidden email]> wrote:
> Getting Jenkins involved in testing isn't the (only) answer, though it
> would certainly help.
>
> If developers who were interested in those databases could watch
> includes/db, that would help, as well.
>

The latter is the real problem here. We don't have any people
who are dedicated to supporting these. People show up, say
they want to work on supporting these, then disappear.

Covering all the non-mysql/sqlite we "support":
- DB2 has been unmaintained for ages, and personally I'm in favor
of dropping that one altogether.
- MSSQL would be nice to improve.
- Oracle support's not bad (maybe not perfect), freakolowsy would
know more.
- Postgres support needs major work. There's a lot of inconsistencies
between PG and the other backends (especially for install/upgrade).
There *are* people here who care about PG.

-Chad

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Mark A. Hershberger-2
On 02/25/2013 10:40 AM, Chad wrote:
> Covering all the non-mysql/sqlite we "support":
> - DB2 has been unmaintained for ages, and personally I'm in favor
> of dropping that one altogether.
> - MSSQL would be nice to improve.
> - Oracle support's not bad (maybe not perfect), freakolowsy would
> know more.
> - Postgres support needs major work. There's a lot of inconsistencies
> between PG and the other backends (especially for install/upgrade).
> There *are* people here who care about PG.

This pretty much mirrors my thinking as well.

Microsoft has been very helpful in the past year by funding some of the
Azure support and MSSQL seems like a natural fit there.  I think we
could get an instance on Azure to help test if we could fit that into
Jenkins somehow.

Other than that, I can only add my +1 to dropping DB2 support.  I don't
know of it being used anywhere.

--
http://hexmode.com/

There is no path to peace. Peace is the path.
   -- Mahatma Gandhi, "Non-Violence in Peace and War"

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Danny Joe Bauch
In reply to this post by Chad
I'm still alive and willing to continue to work on MSSQL.


On Mon, Feb 25, 2013 at 9:40 AM, Chad <[hidden email]> wrote:

> On Mon, Feb 25, 2013 at 7:27 AM, Mark A. Hershberger <[hidden email]>
> wrote:
> > Getting Jenkins involved in testing isn't the (only) answer, though it
> > would certainly help.
> >
> > If developers who were interested in those databases could watch
> > includes/db, that would help, as well.
> >
>
> The latter is the real problem here. We don't have any people
> who are dedicated to supporting these. People show up, say
> they want to work on supporting these, then disappear.
>
> Covering all the non-mysql/sqlite we "support":
> - DB2 has been unmaintained for ages, and personally I'm in favor
> of dropping that one altogether.
> - MSSQL would be nice to improve.
> - Oracle support's not bad (maybe not perfect), freakolowsy would
> know more.
> - Postgres support needs major work. There's a lot of inconsistencies
> between PG and the other backends (especially for install/upgrade).
> There *are* people here who care about PG.
>
> -Chad
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Mark A. Hershberger-2
On 02/25/2013 10:52 AM, Danny Joe Bauch wrote:
> I'm still alive and willing to continue to work on MSSQL.

Have you tried to run MediaWiki's latest from git against MSSQL?


--
http://hexmode.com/

There is no path to peace. Peace is the path.
   -- Mahatma Gandhi, "Non-Violence in Peace and War"

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Danny Joe Bauch
No. The last version I got running was 1.19, and have not worked on any
since. I could do as you suggest, though.


On Mon, Feb 25, 2013 at 9:54 AM, Mark A. Hershberger <[hidden email]>wrote:

> On 02/25/2013 10:52 AM, Danny Joe Bauch wrote:
> > I'm still alive and willing to continue to work on MSSQL.
>
> Have you tried to run MediaWiki's latest from git against MSSQL?
>
>
> --
> http://hexmode.com/
>
> There is no path to peace. Peace is the path.
>    -- Mahatma Gandhi, "Non-Violence in Peace and War"
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Petr Bena
It would be best to have automated environment for this, either on
labs or somewhere else. Problem is that it's not possible to install
MSSQL and such on wikimedia labs given the restrictions

On Mon, Feb 25, 2013 at 5:36 PM, Danny Joe Bauch <[hidden email]> wrote:

> No. The last version I got running was 1.19, and have not worked on any
> since. I could do as you suggest, though.
>
>
> On Mon, Feb 25, 2013 at 9:54 AM, Mark A. Hershberger <[hidden email]>wrote:
>
>> On 02/25/2013 10:52 AM, Danny Joe Bauch wrote:
>> > I'm still alive and willing to continue to work on MSSQL.
>>
>> Have you tried to run MediaWiki's latest from git against MSSQL?
>>
>>
>> --
>> http://hexmode.com/
>>
>> There is no path to peace. Peace is the path.
>>    -- Mahatma Gandhi, "Non-Violence in Peace and War"
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Mark A. Hershberger-2
On 02/25/2013 11:43 AM, Petr Bena wrote:
> It would be best to have automated environment for this, either on
> labs or somewhere else. Problem is that it's not possible to install
> MSSQL and such on wikimedia labs given the restrictions

Which is why I suggested that we work with someone at MS to get an
instance on Azure for testing.


--
http://hexmode.com/

There is no path to peace. Peace is the path.
   -- Mahatma Gandhi, "Non-Violence in Peace and War"

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Danny Joe Bauch
Microsoft was very kind to provide me with an Azure account which I used to
do the testing of 1.18 and 1.19. I've not been in touch with them lately,
nor have I checked whether that account is still open. I would gladly jump
back on that. I'm afraid I got a bit distracted when my job disappeared out
from under me, but I'm back at work at a new job and eager to jump back in
on MSSQL and Azure support.


On Mon, Feb 25, 2013 at 11:04 AM, Mark A. Hershberger <[hidden email]>wrote:

> On 02/25/2013 11:43 AM, Petr Bena wrote:
> > It would be best to have automated environment for this, either on
> > labs or somewhere else. Problem is that it's not possible to install
> > MSSQL and such on wikimedia labs given the restrictions
>
> Which is why I suggested that we work with someone at MS to get an
> instance on Azure for testing.
>
>
> --
> http://hexmode.com/
>
> There is no path to peace. Peace is the path.
>    -- Mahatma Gandhi, "Non-Violence in Peace and War"
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Yuri Astrakhan
In reply to this post by Mark A. Hershberger-2
There are some bugs that also prevents accurate unit testing on multiple
backends (these are the ones I hit personally):

* *Bug 37702* <https://bugzilla.wikimedia.org/show_bug.cgi?id=37702> - Cloned
tables for unittests do not have references and constraints
* *Bug 44790* <https://bugzilla.wikimedia.org/show_bug.cgi?id=44790> - Sqlite
category table contains duplicates in unit tests





On Mon, Feb 25, 2013 at 12:04 PM, Mark A. Hershberger <[hidden email]>wrote:

> On 02/25/2013 11:43 AM, Petr Bena wrote:
> > It would be best to have automated environment for this, either on
> > labs or somewhere else. Problem is that it's not possible to install
> > MSSQL and such on wikimedia labs given the restrictions
>
> Which is why I suggested that we work with someone at MS to get an
> instance on Azure for testing.
>
>
> --
> http://hexmode.com/
>
> There is no path to peace. Peace is the path.
>    -- Mahatma Gandhi, "Non-Violence in Peace and War"
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Sumana Harihareswara-2
In reply to this post by Chad
On 02/25/2013 07:40 AM, Chad wrote:

> On Mon, Feb 25, 2013 at 7:27 AM, Mark A. Hershberger <[hidden email]> wrote:
>> Getting Jenkins involved in testing isn't the (only) answer, though it
>> would certainly help.
>>
>> If developers who were interested in those databases could watch
>> includes/db, that would help, as well.
>>
>
> The latter is the real problem here. We don't have any people
> who are dedicated to supporting these. People show up, say
> they want to work on supporting these, then disappear.
>
> Covering all the non-mysql/sqlite we "support":
> - DB2 has been unmaintained for ages, and personally I'm in favor
> of dropping that one altogether.
> - MSSQL would be nice to improve.
> - Oracle support's not bad (maybe not perfect), freakolowsy would
> know more.
> - Postgres support needs major work. There's a lot of inconsistencies
> between PG and the other backends (especially for install/upgrade).
> There *are* people here who care about PG.
>
> -Chad

Yeah.  Just wanted to point people to prior roundups on this topic --
see
http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/56384
&
https://www.mediawiki.org/wiki/Bug_management/Triage/Databases_20111102
for some people Mark could reach out to to ask for testing & development
help, and for this summary of what needed doing as of November 2011:

> How you can help MediaWiki administrators who don't use MySQL:
>
> * Write tests or specs.  Ben and DJ Bauch want to work on better unit
> testing per https://bugzilla.wikimedia.org/show_bug.cgi?id=32118
> (special page SQL queries), and could use specifications to test
> against.  Improve https://www.mediawiki.org/wiki/Database_testing .

The question: what do we need to test more often to keep RDBMSes happy?
 https://www.mediawiki.org/wiki/Database_testing &
https://www.mediawiki.org/wiki/New_installer/Test_plan Permissions,
searching, schema setup, schema changes, quoting & identifiers, weird
page names, transactions, import, dump, & interface coverage seem like
the main culprits.

And I think we can mostly agree that MySQL/MariaDB (InnoDB & secondarily
MyISAM), SQLite, and PostgreSQL are higher priority than SQL Server/SQL
Server Express, Oracle, and DB2, although others might volunteer as
maintainers and get some of that switched around. :-)

> * Try to reproduce this installation failure on SQLite or PostgreSQL:
> https://bugzilla.wikimedia.org/show_bug.cgi?id=28172

Now fixed.

> * Fix "Database layer should automagically add GROUP BY columns on
> backends that need them (postgres)"
> https://bugzilla.wikimedia.org/show_bug.cgi?id=26273 .  This is a large
> project.

Still needs fixing.

> * Make a meta-schema so that we no longer use tables.sql as a canonical
> source.  Chad and Max started in
> http://svn.wikimedia.org/viewvc/mediawiki/branches/abstract-schema/ .
> See
> https://www.mediawiki.org/wiki/Bug_management/Triage/Databases_20111102#Ideas:
> for more discussion.  This is a large project.

Where's this branch now, if it's still useful?

Thanks for working on this, Mark!

--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Chad
In reply to this post by Mark A. Hershberger-2
On Mon, Feb 25, 2013 at 7:48 AM, Mark A. Hershberger <[hidden email]> wrote:
> Other than that, I can only add my +1 to dropping DB2 support.  I don't
> know of it being used anywhere.
>

https://gerrit.wikimedia.org/r/#/c/50764/

-Chad

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Chad
In reply to this post by Sumana Harihareswara-2
On Mon, Feb 25, 2013 at 9:53 AM, Sumana Harihareswara
<[hidden email]> wrote:
>> * Make a meta-schema so that we no longer use tables.sql as a canonical
>> source.  Chad and Max started in
>> http://svn.wikimedia.org/viewvc/mediawiki/branches/abstract-schema/ .
>> See
>> https://www.mediawiki.org/wiki/Bug_management/Triage/Databases_20111102#Ideas:
>> for more discussion.  This is a large project.
>
> Where's this branch now, if it's still useful?
>

Still in SVN, we never migrated it. It worked as far as installing,
but we never got to the upgrade point. Worth dusting off, but no
way it'll merge cleanly right now.

-Chad

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Freako F. Freakolowsky
In reply to this post by Chad
I'm still active on Oracle front, but as you might have noticed i have
been [WARNING ... understatement ahead] a bit busy in the last 1,5 year
so i just can't manage to pay as much attention to what's going on as i
would like to ...

I used to get paid by my company to do this, but due to some budget cuts
and some other projects that keep me busy at the office and over the
weekends i'm now maintaining this in the little i have left of my own
spare time. I'm still trying to stay up to date, but i just can't manage
to stay involved in the actual development and the comunity as i used
to, but if anyone needs my help with testing and fixing bugs in my
department, my mailbox is always open. Because of this lack of time on
my part i can't keep track of all the changes being made and i would
appretiate it if somone could drop me a line if there are oracle
specific issues ... as i'm doing a lot of DBA work i've got access to
Oracle installations of all shapes and sizes, so i can test on almost
any Oracle version.

As to the current state ... in the last 6 months most of the changes to
the code just was keeping up to date with schema changes ... no major
issues so i'm quite satisfied with the current state, i still wouldn't
suggest it to anyone who would expect it to just plug and play, but if
the admin has any DBA-fu moves up his sleeve it performs nicely. I'm
planning get back into the game and do some development in a month or so
(when i finish one of my other projects) to bring some of the extensions
i've developed for my company and my clients into git and update my
farms to a more recent version (i'm still running those farms on 1.17) ...


... so ... yeah ... i'm still alive and kicking here ... glad to see ppl
still remember i exist :D

LP, Jure


On 25. 02. 2013 16:40, Chad wrote:

> On Mon, Feb 25, 2013 at 7:27 AM, Mark A. Hershberger <[hidden email]> wrote:
>> Getting Jenkins involved in testing isn't the (only) answer, though it
>> would certainly help.
>>
>> If developers who were interested in those databases could watch
>> includes/db, that would help, as well.
>>
> The latter is the real problem here. We don't have any people
> who are dedicated to supporting these. People show up, say
> they want to work on supporting these, then disappear.
>
> Covering all the non-mysql/sqlite we "support":
> - DB2 has been unmaintained for ages, and personally I'm in favor
> of dropping that one altogether.
> - MSSQL would be nice to improve.
> - Oracle support's not bad (maybe not perfect), freakolowsy would
> know more.
> - Postgres support needs major work. There's a lot of inconsistencies
> between PG and the other backends (especially for install/upgrade).
> There *are* people here who care about PG.
>
> -Chad
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Greg Sabino Mullane-2
In reply to this post by Mark A. Hershberger-2
Mark A. Hershberger wrote:
> How can we improve the support for databases like PostgreSQL, Oracle,
> DB2 and MS SQL?
>
> Getting Jenkins involved in testing isn't the (only) answer, though it
> would certainly help.
>
> If developers who were interested in those databases could watch
> includes/db, that would help, as well.

I've tackled this problem, and will share my experience. The problem being how
to keep all the schemas in sync. I looked at the existing solution someone
else started, but found it pretty rough. My idea (I called it "abstract schema")
was a central SQL file, which used a fairly straightforward SQL-ish syntax,
and then a parser that could read that file and create versions for MySQL,
Postgres, etc. Or just load that information into memory for the install
and upgrade process. I actually had a working prototype for that, which
worked quite well. It made the upgrades in particular very smooth, as
there were no more patch files needed, the installer simply read the current
canonical schema state and made the necessary changes.

The big, showstopping problem was trying to map the existing tables.sql
file (the main MySQL one) into the new system. There is no straightforward
mapping possible, as every single table in the system I had to try and
figure out why column such-and-such was using this type, and why sometimes
there was a default and other times not for similar cases, etc. It was quite
the nightmare, so that's why I eventually abandoned the work. The tables.sql
file is quite obviously organically grown, and has little rhyme or reason.
I did come up with some basic naming rules (esp. for indexes), which means
that the first release with the new system will rename a lot of objects,
but as they are not referenced directly, that should not be a problem.

I'm happy to dig out my notes if anyone wants some examples of the
type mapping issues. I think an abstract tables.sql is a good general
approach, but getting from here to there is going to require a lot
of work slogging through those data types.

--
Greg Sabino Mullane [hidden email]
End Point Corporation
PGP Key: 0x14964AC8

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Chad
On Mon, Feb 25, 2013 at 1:29 PM, Greg Sabino Mullane <[hidden email]> wrote:

> Mark A. Hershberger wrote:
>> How can we improve the support for databases like PostgreSQL, Oracle,
>> DB2 and MS SQL?
>>
>> Getting Jenkins involved in testing isn't the (only) answer, though it
>> would certainly help.
>>
>> If developers who were interested in those databases could watch
>> includes/db, that would help, as well.
>
> I've tackled this problem, and will share my experience. The problem being how
> to keep all the schemas in sync. I looked at the existing solution someone
> else started, but found it pretty rough. My idea (I called it "abstract schema")
> was a central SQL file, which used a fairly straightforward SQL-ish syntax,
> and then a parser that could read that file and create versions for MySQL,
> Postgres, etc. Or just load that information into memory for the install
> and upgrade process. I actually had a working prototype for that, which
> worked quite well. It made the upgrades in particular very smooth, as
> there were no more patch files needed, the installer simply read the current
> canonical schema state and made the necessary changes.
>
> The big, showstopping problem was trying to map the existing tables.sql
> file (the main MySQL one) into the new system. There is no straightforward
> mapping possible, as every single table in the system I had to try and
> figure out why column such-and-such was using this type, and why sometimes
> there was a default and other times not for similar cases, etc. It was quite
> the nightmare, so that's why I eventually abandoned the work. The tables.sql
> file is quite obviously organically grown, and has little rhyme or reason.
> I did come up with some basic naming rules (esp. for indexes), which means
> that the first release with the new system will rename a lot of objects,
> but as they are not referenced directly, that should not be a problem.
>
> I'm happy to dig out my notes if anyone wants some examples of the
> type mapping issues. I think an abstract tables.sql is a good general
> approach, but getting from here to there is going to require a lot
> of work slogging through those data types.
>

This was exactly what we tried with the abstract-schema branch,
before it was abandoned. The place we got stuck on was handling
updates.

-Chad

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Tyler Romeo
Where is the abstract-schema branch now? The only thing I remember about
that was a brief mailing list discussion (I think started by Daniel
Friesnen) about making such a format. What exactly were the hold-ups with
updates?

*--*
*Tyler Romeo*
Stevens Institute of Technology, Class of 2015
Major in Computer Science
www.whizkidztech.com | [hidden email]


On Mon, Feb 25, 2013 at 4:35 PM, Chad <[hidden email]> wrote:

> On Mon, Feb 25, 2013 at 1:29 PM, Greg Sabino Mullane <[hidden email]>
> wrote:
> > Mark A. Hershberger wrote:
> >> How can we improve the support for databases like PostgreSQL, Oracle,
> >> DB2 and MS SQL?
> >>
> >> Getting Jenkins involved in testing isn't the (only) answer, though it
> >> would certainly help.
> >>
> >> If developers who were interested in those databases could watch
> >> includes/db, that would help, as well.
> >
> > I've tackled this problem, and will share my experience. The problem
> being how
> > to keep all the schemas in sync. I looked at the existing solution
> someone
> > else started, but found it pretty rough. My idea (I called it "abstract
> schema")
> > was a central SQL file, which used a fairly straightforward SQL-ish
> syntax,
> > and then a parser that could read that file and create versions for
> MySQL,
> > Postgres, etc. Or just load that information into memory for the install
> > and upgrade process. I actually had a working prototype for that, which
> > worked quite well. It made the upgrades in particular very smooth, as
> > there were no more patch files needed, the installer simply read the
> current
> > canonical schema state and made the necessary changes.
> >
> > The big, showstopping problem was trying to map the existing tables.sql
> > file (the main MySQL one) into the new system. There is no
> straightforward
> > mapping possible, as every single table in the system I had to try and
> > figure out why column such-and-such was using this type, and why
> sometimes
> > there was a default and other times not for similar cases, etc. It was
> quite
> > the nightmare, so that's why I eventually abandoned the work. The
> tables.sql
> > file is quite obviously organically grown, and has little rhyme or
> reason.
> > I did come up with some basic naming rules (esp. for indexes), which
> means
> > that the first release with the new system will rename a lot of objects,
> > but as they are not referenced directly, that should not be a problem.
> >
> > I'm happy to dig out my notes if anyone wants some examples of the
> > type mapping issues. I think an abstract tables.sql is a good general
> > approach, but getting from here to there is going to require a lot
> > of work slogging through those data types.
> >
>
> This was exactly what we tried with the abstract-schema branch,
> before it was abandoned. The place we got stuck on was handling
> updates.
>
> -Chad
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Better non-MySQL db support

Chad
On Mon, Feb 25, 2013 at 2:00 PM, Tyler Romeo <[hidden email]> wrote:
> Where is the abstract-schema branch now? The only thing I remember about
> that was a brief mailing list discussion (I think started by Daniel
> Friesnen) about making such a format. What exactly were the hold-ups with
> updates?
>

As I said earlier in the thread, it's still in SVN. I didn't bother migrating
it because nobody was caring at the time. If anyone's wanting to dust
it off and make a new branch for core, we can do that.

-Chad

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
12