many runjobs

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

many runjobs

Martin Brunner
Hi!

I'm running a SMW with about 7000 pages.

If I start the runjob.php (in the maintenance directory on the server),
I'll have about 7000 jobs after a day.

It seems that per minute, 5 new jobs are created for some reason. Is
that normal?

The jobs that are running, are like:
2016-09-28 19:16:31 refreshLinksDynamic [Pagetitle] isOpportunistic=1
rootJobTimestamp=20160928191530 requestId=[requestId]
(id=[id],timestamp=[timestamp]) STARTING

(where text in [] contains some names and numbers)

Regards
Martin

------------------------------------------------------------------------------
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

Martin Brunner
No one who has any idea?

It seems that these jobs are done permanently, but i wonder if this is
really necesary or if I can so something to avoid these runjobs and
server load.

Martin Brunner schrieb:

> Hi!
>
> I'm running a SMW with about 7000 pages.
>
> If I start the runjob.php (in the maintenance directory on the
> server), I'll have about 7000 jobs after a day.
>
> It seems that per minute, 5 new jobs are created for some reason. Is
> that normal?
>
> The jobs that are running, are like: 2016-09-28 19:16:31
> refreshLinksDynamic [Pagetitle] isOpportunistic=1
> rootJobTimestamp=20160928191530 requestId=[requestId]
> (id=[id],timestamp=[timestamp]) STARTING
>
> (where text in [] contains some names and numbers)
>
> Regards Martin
>
> ------------------------------------------------------------------------------
>
>
_______________________________________________
> Semediawiki-user mailing list [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

James HK
Hi,

>> refreshLinksDynamic [Pagetitle] isOpportunistic=1
>> rootJobTimestamp=20160928191530 requestId=[requestId]
>> (id=[id],timestamp=[timestamp]) STARTING

The mentioned `refreshLinksDynamic` is created by [0, 1] with "...
update links for backlink pages..." in case a template or other
related objects are altered that belong to a page.

[0] https://github.com/wikimedia/mediawiki/blob/ec7d855d421646e9dd9dbe2254a58f8a85111e0a/includes/jobqueue/jobs/RefreshLinksJob.php#L25-L38

[1] https://www.mediawiki.org/wiki/Manual:RefreshLinksJob.php

Cheers

On 10/5/16, Martin Brunner <[hidden email]> wrote:

> No one who has any idea?
>
> It seems that these jobs are done permanently, but i wonder if this is
> really necesary or if I can so something to avoid these runjobs and
> server load.
>
> Martin Brunner schrieb:
>> Hi!
>>
>> I'm running a SMW with about 7000 pages.
>>
>> If I start the runjob.php (in the maintenance directory on the
>> server), I'll have about 7000 jobs after a day.
>>
>> It seems that per minute, 5 new jobs are created for some reason. Is
>> that normal?
>>
>> The jobs that are running, are like: 2016-09-28 19:16:31
>> refreshLinksDynamic [Pagetitle] isOpportunistic=1
>> rootJobTimestamp=20160928191530 requestId=[requestId]
>> (id=[id],timestamp=[timestamp]) STARTING
>>
>> (where text in [] contains some names and numbers)
>>
>> Regards Martin
>>
>> ------------------------------------------------------------------------------
>>
>>
> _______________________________________________
>> Semediawiki-user mailing list [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

Martin Brunner
James HK schrieb:

> Hi,
>
>>> refreshLinksDynamic [Pagetitle] isOpportunistic=1
>>> rootJobTimestamp=20160928191530 requestId=[requestId]
>>> (id=[id],timestamp=[timestamp]) STARTING
>
> The mentioned `refreshLinksDynamic` is created by [0, 1] with "...
> update links for backlink pages..." in case a template or other
> related objects are altered that belong to a page.
>
> [0] https://github.com/wikimedia/mediawiki/blob/ec7d855d421646e9dd9dbe2254a58f8a85111e0a/includes/jobqueue/jobs/RefreshLinksJob.php#L25-L38
>
> [1] https://www.mediawiki.org/wiki/Manual:RefreshLinksJob.php

Ah ok. But still I wonder, why this also happens permanentely even if we didn't
change anything on the site.

There are also other RunJobs that are caused by SMW rather than MW, especially
when it comes to Attributes. But it seems that these MW-runjobs are blocking the
SMW runjobs we need to be done urgently.

Regards
Martin


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

James HK
Hi,

> There are also other RunJobs that are caused by SMW rather than MW,
> especially

If you change the type (or other specification details) of a property
then SMW will schedule a [0]. Also, if certain updates are blocked
during the deferred update process (2.4+) then a UpdateJob is
scheduled to make necessary secondary updates when available.

> when it comes to Attributes. But it seems that these MW-runjobs are blocking
> the
> SMW runjobs we need to be done urgently.

As far as I know (not that I really looked into it), the JobScheduler
works sequential (on the FIFO principle) and I'm not aware of allowing
specific jobs to be scheduled with a higher priority.

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/src/MediaWiki/Jobs/UpdateDispatcherJob.php

Cheers

On 10/5/16, Martin Brunner <[hidden email]> wrote:

> James HK schrieb:
>> Hi,
>>
>>>> refreshLinksDynamic [Pagetitle] isOpportunistic=1
>>>> rootJobTimestamp=20160928191530 requestId=[requestId]
>>>> (id=[id],timestamp=[timestamp]) STARTING
>>
>> The mentioned `refreshLinksDynamic` is created by [0, 1] with "...
>> update links for backlink pages..." in case a template or other
>> related objects are altered that belong to a page.
>>
>> [0]
>> https://github.com/wikimedia/mediawiki/blob/ec7d855d421646e9dd9dbe2254a58f8a85111e0a/includes/jobqueue/jobs/RefreshLinksJob.php#L25-L38
>>
>> [1] https://www.mediawiki.org/wiki/Manual:RefreshLinksJob.php
>
> Ah ok. But still I wonder, why this also happens permanentely even if we
> didn't
> change anything on the site.
>
> There are also other RunJobs that are caused by SMW rather than MW,
> especially
> when it comes to Attributes. But it seems that these MW-runjobs are blocking
> the
> SMW runjobs we need to be done urgently.
>
> Regards
> Martin
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

Martin Brunner
James HK schrieb:
> As far as I know (not that I really looked into it), the JobScheduler
> works sequential (on the FIFO principle) and I'm not aware of allowing
> specific jobs to be scheduled with a higher priority.

The problem we have:

We are using a certain attribute to specify which Semantic Forms
formular we are using. E.g. if the entry has an attribute like "address"
then we provide a formular for editing the address.

The problems is: If you create a new page with this formular, the job
will processed too late. So in case you want do edit something again,
you have to wait half a day until the formular button appears.

If we set the $wgJobRunRate to 1, then we'll have 9.000 Jobs in 10
minutes which seems to be too much for the server...

So, I don't know what to do to improve this... :(

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

kghbln
Why don't you use cronjobs to clear your job queue like e.g "php
runJobs.php --quiet --maxjobs=500" and do this every three minutes or so.

A long time ago I stopped relying on page accesses and edits to clear
this. Also you could consider using redis to move the jobqueue away from
the database.

Both MediaWiki and Semantic MediaWiki increasingly move stuff to the job
queue to get things done.

Cheers Karsten

Am 08.10.2016 um 19:06 schrieb Martin Brunner:

> James HK schrieb:
>> As far as I know (not that I really looked into it), the JobScheduler
>> works sequential (on the FIFO principle) and I'm not aware of allowing
>> specific jobs to be scheduled with a higher priority.
>
> The problem we have:
>
> We are using a certain attribute to specify which Semantic Forms
> formular we are using. E.g. if the entry has an attribute like "address"
> then we provide a formular for editing the address.
>
> The problems is: If you create a new page with this formular, the job
> will processed too late. So in case you want do edit something again,
> you have to wait half a day until the formular button appears.
>
> If we set the $wgJobRunRate to 1, then we'll have 9.000 Jobs in 10
> minutes which seems to be too much for the server...
>
> So, I don't know what to do to improve this... :(
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

Martin Brunner
kghbln schrieb:
> Why don't you use cronjobs to clear your job queue like e.g "php
> runJobs.php --quiet --maxjobs=500" and do this every three minutes or
> so.

Yes, I was thinking about doing so, but the server is too weak for
maxjobs=500 all three minutes...

I still wonder where all the jobs come from and if it is possible to
reduce them.

Cheers,
Martin

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

John Doe-27
Reducing job queue has a negative impact on page load/save times. Play with
both the maxjobs and run intervals to see what works for your wiki.

On Saturday, October 8, 2016, Martin Brunner <[hidden email]> wrote:

> kghbln schrieb:
> > Why don't you use cronjobs to clear your job queue like e.g "php
> > runJobs.php --quiet --maxjobs=500" and do this every three minutes or
> > so.
>
> Yes, I was thinking about doing so, but the server is too weak for
> maxjobs=500 all three minutes...
>
> I still wonder where all the jobs come from and if it is possible to
> reduce them.
>
> Cheers,
> Martin
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email] <javascript:;>
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

James HK
In reply to this post by Martin Brunner
Hi,

> If we set the $wgJobRunRate to 1, then we'll have 9.000 Jobs in 10
> minutes which seems to be too much for the server...

It depends on the on the amount of users who do changes to pages or
templates but somehow I find that quite a lot (especially given the
rate per minute).

Cheers

On 10/8/16, Martin Brunner <[hidden email]> wrote:

> James HK schrieb:
>> As far as I know (not that I really looked into it), the JobScheduler
>> works sequential (on the FIFO principle) and I'm not aware of allowing
>> specific jobs to be scheduled with a higher priority.
>
> The problem we have:
>
> We are using a certain attribute to specify which Semantic Forms
> formular we are using. E.g. if the entry has an attribute like "address"
> then we provide a formular for editing the address.
>
> The problems is: If you create a new page with this formular, the job
> will processed too late. So in case you want do edit something again,
> you have to wait half a day until the formular button appears.
>
> If we set the $wgJobRunRate to 1, then we'll have 9.000 Jobs in 10
> minutes which seems to be too much for the server...
>
> So, I don't know what to do to improve this... :(
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

James HK
In reply to this post by kghbln
Hi,

> Both MediaWiki and Semantic MediaWiki increasingly move stuff to the job
> queue to get things done.

Well, MediaWiki started to use deferred updates extensively with 1.27+
[0] and Semantic MediaWiki just tries to keep up.

Updates that are required are posted as transaction and processed
within the same queue as updates to a wikipage but secondary updates
are either deferred to a different socket request or if the web server
is too busy and cannot be reached those updates are registered as
jobs.

> this. Also you could consider using redis to move the jobqueue away from

Yes, and as noted somewhere on smw@wiki; try to encapsulate DB
operations from secondary read/write operations by shifting work to a
redis instance. redis as a key/value store has shown good performance
for various tasks and our recommendation is to use it whenever
possible (as cache instance etc.).

PS: To avoid confusion, the SQLStore remains a SQL instance.

[0] https://www.semantic-mediawiki.org/wiki/Deferred_updates

Cheers

On 10/8/16, kghbln <[hidden email]> wrote:

> Why don't you use cronjobs to clear your job queue like e.g "php
> runJobs.php --quiet --maxjobs=500" and do this every three minutes or so.
>
> A long time ago I stopped relying on page accesses and edits to clear
> this. Also you could consider using redis to move the jobqueue away from
> the database.
>
> Both MediaWiki and Semantic MediaWiki increasingly move stuff to the job
> queue to get things done.
>
> Cheers Karsten
>
> Am 08.10.2016 um 19:06 schrieb Martin Brunner:
>> James HK schrieb:
>>> As far as I know (not that I really looked into it), the JobScheduler
>>> works sequential (on the FIFO principle) and I'm not aware of allowing
>>> specific jobs to be scheduled with a higher priority.
>>
>> The problem we have:
>>
>> We are using a certain attribute to specify which Semantic Forms
>> formular we are using. E.g. if the entry has an attribute like "address"
>> then we provide a formular for editing the address.
>>
>> The problems is: If you create a new page with this formular, the job
>> will processed too late. So in case you want do edit something again,
>> you have to wait half a day until the formular button appears.
>>
>> If we set the $wgJobRunRate to 1, then we'll have 9.000 Jobs in 10
>> minutes which seems to be too much for the server...
>>
>> So, I don't know what to do to improve this... :(
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Semediawiki-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: many runjobs

Michael Erdmann
In reply to this post by Martin Brunner
Hi Martin,
I cannot tell why you have that many jobs in the queue but here is a
small script we use to keep the queue empty by executing all jobs more
or less permanently but not within a HTTP request but scheduled by CRON.

Set
     $wgJobRunRate = 0;
in your LocalSettings.php

create runJobs.sh
     #!/usr/bin/env bash
     # will be executed by a cron-job
     MEDIAWIKI=/path/to/your/mediawiki
     RUNNING=$(ps aux | grep /mediawiki/maintenance/runJobs.php | grep
maxjobs)
     if [ -z $RUNNING ]; then su XXX -c "php
$MEDIAWIKI/maintenance/runJobs.php --maxjobs=999" ;fi

Set XXX to a user that should execute runJobs.php. And have CRON execute
the shell-script every minute or so.

HTH,
   michael

On 28.09.2016 21:22, Martin Brunner wrote:

> Hi!
>
> I'm running a SMW with about 7000 pages.
>
> If I start the runjob.php (in the maintenance directory on the server),
> I'll have about 7000 jobs after a day.
>
> It seems that per minute, 5 new jobs are created for some reason. Is
> that normal?
>
> The jobs that are running, are like:
> 2016-09-28 19:16:31 refreshLinksDynamic [Pagetitle] isOpportunistic=1
> rootJobTimestamp=20160928191530 requestId=[requestId]
> (id=[id],timestamp=[timestamp]) STARTING
>
> (where text in [] contains some names and numbers)
>
> Regards
> Martin
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user


--

Dr. Michael Erdmann    |    [hidden email]    |   +49 151 6140 1790
DIQA Projektmanagement GmbH | Pfinztalstr. 90 | 76227 Karlsruhe, Germany
Handelsregister: Amtsgericht Mannheim HRB 715454 | USt-IdNr: DE283037270
Geschäftsführer: Dr. Michael Erdmann,  Dipl.-Wirtsch.-Inf. Daniel Hansch

This email may contain confidential information. If you are not the
intended recipient please notify the sender immediately and delete this
email. Any unauthorized copying, disclosure or distribution of this
email is strictly forbidden.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user