SMW UpdateJob error

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

SMW UpdateJob error

Justin Lloyd-2
Hi all,

Sorry in advance this is so long, but I've got a production issue that has
me completely stumped and I wanted to put all information I have out there.

I've asked about this on the IRC channel but wasn't able to determine the
cause of the issue. Basically, I'm getting intermittent fatal MWExceptions
on one of my live wikis, which started Monday afternoon. I have five wikis
and this is the only one affected, all are on the same versions of
everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
issue, having just started Monday afternoon and nothing at the server or
application level has changed. It's possible that something in the content
has changed, though there's nothing I can see in the recent changes log
that might be a culprit (just some very basic page creating and edit
testing), though I'm not an SMW expert, so I don't know for sure.

I can't hit the wiki normally so when I hostfile to access one of the
server farm IPs directly, I can hit it just fine and refreshing it, I see
the MWException message embedded in pages intermittently, even when just
refreshing the same page. Enabling the MW debug log, the difference I see
between successful and unsuccessful page loads is this:

DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
[76103ff0] /wiki/Pagenamehere   Exception from line 225 of
/path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
#0 /path/to/wiki/includes/job/Job.php(200): Job::factory('SMW\UpdateJob',
Object(Title), false, '1')
#1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
#2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
#3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
#4 /path/to/wiki/index.php(59): MediaWiki->run()
#5 {main}

I found this discussion on the SMW Installation talk page
<https://semantic-mediawiki.org/wiki/Thread:Help_talk:Installation/Re:_Warning:_Class_'%E2%80%A6'_not_found_in_%E2%80%A6/reply>
but
that did not help, though it does show the call using SMWUpdateJob rather
than SMW\UpdateJob. I found this code
<http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that shows
the UpdateJob constructor using the backslash but that's not what our code
contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
parent::__construct( 'SMWUpdateJob', $title ) in the constructor.

I've dug everywhere I can think of, including running the strings command
against all of the database files, but I cannot for the life of me figure
out where this occasional incorrect parameter is coming from. There are no
jobs in the database (it's a low-traffic wiki) and even refreshing a lot
and watching the table I wasn't seeing and jobs appear. Nothing has changed
on the server (I was even out of the office yesterday and I'm the only one
that touches the wikis).

This intermittent issue happens enough that Varnish is receiving 500s from
Apache for the wiki's vhost, causing the load balancer in front of Varnish
to return 502s to clients, so the live wiki is down.

Any advice is welcome!

Justin
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce.
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: SMW UpdateJob error

James HK
Hi,

> and this is the only one affected, all are on the same versions of
> everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
> issue, having just started Monday afternoon and nothing at the server or

> DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
> [76103ff0] /wiki/Pagenamehere   Exception from line 225 of
> /path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
> #0 /path/to/wiki/includes/job/Job.php(200): Job::factory('SMW\UpdateJob',
> Object(Title), false, '1')
> #1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
> #2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
> #3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
> #4 /path/to/wiki/index.php(59): MediaWiki->run()
> #5 {main}

MW tries to delete a job row with the name of `SMW\UpdateJob` and
fails which shouldn't/can't happen on a SMW 1.8.0.5 system because the
`SMW\UpdateJob` definition was only introduced with SMW 1.9 (see [0,
1]).

> that did not help, though it does show the call using SMWUpdateJob rather
> than SMW\UpdateJob. I found this code
> <http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that shows
> the UpdateJob constructor using the backslash but that's not what our code
> contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
> parent::__construct( 'SMWUpdateJob', $title ) in the constructor.

SMW\UpdateJob is only relevant to SMW 1.9+.

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.8.0.5/includes/jobs/SMW_UpdateJob.php

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.9/includes/jobs/UpdateJob.php

I have no idea why your 1.8.0.5 is referencing to `SMW\UpdateJob` but
to avoid a failure you should delete all jobs from the queue with the
name of `SMW\UpdateJob`.

Cheers

On 9/17/14, Justin Lloyd <[hidden email]> wrote:

> Hi all,
>
> Sorry in advance this is so long, but I've got a production issue that has
> me completely stumped and I wanted to put all information I have out there.
>
> I've asked about this on the IRC channel but wasn't able to determine the
> cause of the issue. Basically, I'm getting intermittent fatal MWExceptions
> on one of my live wikis, which started Monday afternoon. I have five wikis
> and this is the only one affected, all are on the same versions of
> everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
> issue, having just started Monday afternoon and nothing at the server or
> application level has changed. It's possible that something in the content
> has changed, though there's nothing I can see in the recent changes log
> that might be a culprit (just some very basic page creating and edit
> testing), though I'm not an SMW expert, so I don't know for sure.
>
> I can't hit the wiki normally so when I hostfile to access one of the
> server farm IPs directly, I can hit it just fine and refreshing it, I see
> the MWException message embedded in pages intermittently, even when just
> refreshing the same page. Enabling the MW debug log, the difference I see
> between successful and unsuccessful page loads is this:
>
> DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
> [76103ff0] /wiki/Pagenamehere   Exception from line 225 of
> /path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
> #0 /path/to/wiki/includes/job/Job.php(200): Job::factory('SMW\UpdateJob',
> Object(Title), false, '1')
> #1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
> #2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
> #3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
> #4 /path/to/wiki/index.php(59): MediaWiki->run()
> #5 {main}
>
> I found this discussion on the SMW Installation talk page
> <https://semantic-mediawiki.org/wiki/Thread:Help_talk:Installation/Re:_Warning:_Class_'%E2%80%A6'_not_found_in_%E2%80%A6/reply>
> but
> that did not help, though it does show the call using SMWUpdateJob rather
> than SMW\UpdateJob. I found this code
> <http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that shows
> the UpdateJob constructor using the backslash but that's not what our code
> contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
> parent::__construct( 'SMWUpdateJob', $title ) in the constructor.
>
> I've dug everywhere I can think of, including running the strings command
> against all of the database files, but I cannot for the life of me figure
> out where this occasional incorrect parameter is coming from. There are no
> jobs in the database (it's a low-traffic wiki) and even refreshing a lot
> and watching the table I wasn't seeing and jobs appear. Nothing has changed
> on the server (I was even out of the office yesterday and I'm the only one
> that touches the wikis).
>
> This intermittent issue happens enough that Varnish is receiving 500s from
> Apache for the wiki's vhost, causing the load balancer in front of Varnish
> to return 502s to clients, so the live wiki is down.
>
> Any advice is welcome!
>
> Justin
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce.
> Perforce version control. Predictably reliable.
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: SMW UpdateJob error

Justin Lloyd-2
Hi James,

So that somewhat explains the different string but I don't know why it's
ended up in the live database. That being said, the job table is empty, so
I don't really know what's happening. Perhaps jobs are getting created and
deleted in very short order, with virtually every page view, yet still
somehow yielding the MWException. Any ideas on what could be creating jobs
with the incorrect string?

I did some maintenance last night (restarting Apache, Varnish, and
Memcached on all of the web servers), and though it didn't resolve the
problem (obviously, based on your explanation) one of the four-load
balanced servers is seen as healthy (enough) by the load balancer so the
wiki is visible (for the time being, at least), but still showing
MWException errors with almost every page view. I even tried
SMW_refreshData.php, no luck there either.

Thanks,
Justin


On Wed, Sep 17, 2014 at 3:00 AM, James HK <[hidden email]>
wrote:

> Hi,
>
> > and this is the only one affected, all are on the same versions of
> > everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
> > issue, having just started Monday afternoon and nothing at the server or
>
> > DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
> > [76103ff0] /wiki/Pagenamehere   Exception from line 225 of
> > /path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
> > #0 /path/to/wiki/includes/job/Job.php(200): Job::factory('SMW\UpdateJob',
> > Object(Title), false, '1')
> > #1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
> > #2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
> > #3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
> > #4 /path/to/wiki/index.php(59): MediaWiki->run()
> > #5 {main}
>
> MW tries to delete a job row with the name of `SMW\UpdateJob` and
> fails which shouldn't/can't happen on a SMW 1.8.0.5 system because the
> `SMW\UpdateJob` definition was only introduced with SMW 1.9 (see [0,
> 1]).
>
> > that did not help, though it does show the call using SMWUpdateJob rather
> > than SMW\UpdateJob. I found this code
> > <http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that
> shows
> > the UpdateJob constructor using the backslash but that's not what our
> code
> > contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
> > parent::__construct( 'SMWUpdateJob', $title ) in the constructor.
>
> SMW\UpdateJob is only relevant to SMW 1.9+.
>
> [0]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.8.0.5/includes/jobs/SMW_UpdateJob.php
>
> [1]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.9/includes/jobs/UpdateJob.php
>
> I have no idea why your 1.8.0.5 is referencing to `SMW\UpdateJob` but
> to avoid a failure you should delete all jobs from the queue with the
> name of `SMW\UpdateJob`.
>
> Cheers
>
> On 9/17/14, Justin Lloyd <[hidden email]> wrote:
> > Hi all,
> >
> > Sorry in advance this is so long, but I've got a production issue that
> has
> > me completely stumped and I wanted to put all information I have out
> there.
> >
> > I've asked about this on the IRC channel but wasn't able to determine the
> > cause of the issue. Basically, I'm getting intermittent fatal
> MWExceptions
> > on one of my live wikis, which started Monday afternoon. I have five
> wikis
> > and this is the only one affected, all are on the same versions of
> > everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
> > issue, having just started Monday afternoon and nothing at the server or
> > application level has changed. It's possible that something in the
> content
> > has changed, though there's nothing I can see in the recent changes log
> > that might be a culprit (just some very basic page creating and edit
> > testing), though I'm not an SMW expert, so I don't know for sure.
> >
> > I can't hit the wiki normally so when I hostfile to access one of the
> > server farm IPs directly, I can hit it just fine and refreshing it, I see
> > the MWException message embedded in pages intermittently, even when just
> > refreshing the same page. Enabling the MW debug log, the difference I see
> > between successful and unsuccessful page loads is this:
> >
> > DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
> > [76103ff0] /wiki/Pagenamehere   Exception from line 225 of
> > /path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
> > #0 /path/to/wiki/includes/job/Job.php(200): Job::factory('SMW\UpdateJob',
> > Object(Title), false, '1')
> > #1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
> > #2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
> > #3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
> > #4 /path/to/wiki/index.php(59): MediaWiki->run()
> > #5 {main}
> >
> > I found this discussion on the SMW Installation talk page
> > <
> https://semantic-mediawiki.org/wiki/Thread:Help_talk:Installation/Re:_Warning:_Class_'%E2%80%A6'_not_found_in_%E2%80%A6/reply
> >
> > but
> > that did not help, though it does show the call using SMWUpdateJob rather
> > than SMW\UpdateJob. I found this code
> > <http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that
> shows
> > the UpdateJob constructor using the backslash but that's not what our
> code
> > contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
> > parent::__construct( 'SMWUpdateJob', $title ) in the constructor.
> >
> > I've dug everywhere I can think of, including running the strings command
> > against all of the database files, but I cannot for the life of me figure
> > out where this occasional incorrect parameter is coming from. There are
> no
> > jobs in the database (it's a low-traffic wiki) and even refreshing a lot
> > and watching the table I wasn't seeing and jobs appear. Nothing has
> changed
> > on the server (I was even out of the office yesterday and I'm the only
> one
> > that touches the wikis).
> >
> > This intermittent issue happens enough that Varnish is receiving 500s
> from
> > Apache for the wiki's vhost, causing the load balancer in front of
> Varnish
> > to return 502s to clients, so the live wiki is down.
> >
> > Any advice is welcome!
> >
> > Justin
> >
> ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce.
> > Perforce version control. Predictably reliable.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Semediawiki-user mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/semediawiki-user
> >
>
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: SMW UpdateJob error

Justin Lloyd-2
I've figured out the problem and it was actually due to a misconfiguration.

Justin


On Wed, Sep 17, 2014 at 10:46 AM, Justin Lloyd <[hidden email]> wrote:

> Hi James,
>
> So that somewhat explains the different string but I don't know why it's
> ended up in the live database. That being said, the job table is empty, so
> I don't really know what's happening. Perhaps jobs are getting created and
> deleted in very short order, with virtually every page view, yet still
> somehow yielding the MWException. Any ideas on what could be creating jobs
> with the incorrect string?
>
> I did some maintenance last night (restarting Apache, Varnish, and
> Memcached on all of the web servers), and though it didn't resolve the
> problem (obviously, based on your explanation) one of the four-load
> balanced servers is seen as healthy (enough) by the load balancer so the
> wiki is visible (for the time being, at least), but still showing
> MWException errors with almost every page view. I even tried
> SMW_refreshData.php, no luck there either.
>
> Thanks,
> Justin
>
>
> On Wed, Sep 17, 2014 at 3:00 AM, James HK <[hidden email]>
> wrote:
>
>> Hi,
>>
>> > and this is the only one affected, all are on the same versions of
>> > everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
>> > issue, having just started Monday afternoon and nothing at the server or
>>
>> > DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
>> > [76103ff0] /wiki/Pagenamehere   Exception from line 225 of
>> > /path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
>> > #0 /path/to/wiki/includes/job/Job.php(200):
>> Job::factory('SMW\UpdateJob',
>> > Object(Title), false, '1')
>> > #1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
>> > #2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
>> > #3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
>> > #4 /path/to/wiki/index.php(59): MediaWiki->run()
>> > #5 {main}
>>
>> MW tries to delete a job row with the name of `SMW\UpdateJob` and
>> fails which shouldn't/can't happen on a SMW 1.8.0.5 system because the
>> `SMW\UpdateJob` definition was only introduced with SMW 1.9 (see [0,
>> 1]).
>>
>> > that did not help, though it does show the call using SMWUpdateJob
>> rather
>> > than SMW\UpdateJob. I found this code
>> > <http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that
>> shows
>> > the UpdateJob constructor using the backslash but that's not what our
>> code
>> > contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
>> > parent::__construct( 'SMWUpdateJob', $title ) in the constructor.
>>
>> SMW\UpdateJob is only relevant to SMW 1.9+.
>>
>> [0]
>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.8.0.5/includes/jobs/SMW_UpdateJob.php
>>
>> [1]
>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.9/includes/jobs/UpdateJob.php
>>
>> I have no idea why your 1.8.0.5 is referencing to `SMW\UpdateJob` but
>> to avoid a failure you should delete all jobs from the queue with the
>> name of `SMW\UpdateJob`.
>>
>> Cheers
>>
>> On 9/17/14, Justin Lloyd <[hidden email]> wrote:
>> > Hi all,
>> >
>> > Sorry in advance this is so long, but I've got a production issue that
>> has
>> > me completely stumped and I wanted to put all information I have out
>> there.
>> >
>> > I've asked about this on the IRC channel but wasn't able to determine
>> the
>> > cause of the issue. Basically, I'm getting intermittent fatal
>> MWExceptions
>> > on one of my live wikis, which started Monday afternoon. I have five
>> wikis
>> > and this is the only one affected, all are on the same versions of
>> > everything (MW 1.20.8, SMW 1.8.0.5, PHP 5.3.10, etc.). This is a recent
>> > issue, having just started Monday afternoon and nothing at the server or
>> > application level has changed. It's possible that something in the
>> content
>> > has changed, though there's nothing I can see in the recent changes log
>> > that might be a culprit (just some very basic page creating and edit
>> > testing), though I'm not an SMW expert, so I don't know for sure.
>> >
>> > I can't hit the wiki normally so when I hostfile to access one of the
>> > server farm IPs directly, I can hit it just fine and refreshing it, I
>> see
>> > the MWException message embedded in pages intermittently, even when just
>> > refreshing the same page. Enabling the MW debug log, the difference I
>> see
>> > between successful and unsuccessful page loads is this:
>> >
>> > DatabaseBase::query: Writes done: DELETE FROM `job` WHERE job_id = '1'
>> > [76103ff0] /wiki/Pagenamehere   Exception from line 225 of
>> > /path/to/wiki/includes/job/Job.php: Invalid job command `SMW\UpdateJob`
>> > #0 /path/to/wiki/includes/job/Job.php(200):
>> Job::factory('SMW\UpdateJob',
>> > Object(Title), false, '1')
>> > #1 /path/to/wiki/includes/Wiki.php(591): Job::pop()
>> > #2 /path/to/wiki/includes/Wiki.php(556): MediaWiki->doJobs()
>> > #3 /path/to/wiki/includes/Wiki.php(447): MediaWiki->restInPeace()
>> > #4 /path/to/wiki/index.php(59): MediaWiki->run()
>> > #5 {main}
>> >
>> > I found this discussion on the SMW Installation talk page
>> > <
>> https://semantic-mediawiki.org/wiki/Thread:Help_talk:Installation/Re:_Warning:_Class_'%E2%80%A6'_not_found_in_%E2%80%A6/reply
>> >
>> > but
>> > that did not help, though it does show the call using SMWUpdateJob
>> rather
>> > than SMW\UpdateJob. I found this code
>> > <http://semantic-mediawiki.org/doc/UpdateJob_8php_source.html> that
>> shows
>> > the UpdateJob constructor using the backslash but that's not what our
>> code
>> > contains. Rather, SMW/includes/jobs/SMW_UpdateJob.php does contain
>> > parent::__construct( 'SMWUpdateJob', $title ) in the constructor.
>> >
>> > I've dug everywhere I can think of, including running the strings
>> command
>> > against all of the database files, but I cannot for the life of me
>> figure
>> > out where this occasional incorrect parameter is coming from. There are
>> no
>> > jobs in the database (it's a low-traffic wiki) and even refreshing a lot
>> > and watching the table I wasn't seeing and jobs appear. Nothing has
>> changed
>> > on the server (I was even out of the office yesterday and I'm the only
>> one
>> > that touches the wikis).
>> >
>> > This intermittent issue happens enough that Varnish is receiving 500s
>> from
>> > Apache for the wiki's vhost, causing the load balancer in front of
>> Varnish
>> > to return 502s to clients, so the live wiki is down.
>> >
>> > Any advice is welcome!
>> >
>> > Justin
>> >
>> ------------------------------------------------------------------------------
>> > Want excitement?
>> > Manually upgrade your production database.
>> > When you want reliability, choose Perforce.
>> > Perforce version control. Predictably reliable.
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>> > _______________________________________________
>> > Semediawiki-user mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>> >
>>
>
>
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user