[MediaWiki-l] Mediawiki Job queue not running

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[MediaWiki-l] Mediawiki Job queue not running

Andrew Geary
I administer a public wiki - https://wiki.seg.org - and I am having issues
with a jobs backlog. It's currently at 19,058 and counting. I have tried
using runJobs.php, but I get this error message:

PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to
allocate 3846541 bytes) in
/var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656

I have noticed I do not have the variable:

$wgJobRunRate

listed in my LocalSettings.php.

Should I add this variable to my LocalSettings.php file? How can I address
this error message? I believe the jobs backlog is contributing to a
Translate extension error I am having, mentioned here -
https://phabricator.wikimedia.org/T159947.

I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
No changes have been made to my version of Mediawiki in that timeframe.
Thank you.

Best,
Andrew

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA


_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Greg Rundlett (freephile)
Hi Andrew,

As the error says, you're running out of memory on the execution of
runJobs.php

Try

php maintenance/runJobs.php --memory-limit max --maxjobs 5

If that completes without error, then try increasing the number of jobs
allowed with --maxjobs

Greg Rundlett
https://eQuality-Tech.com
https://freephile.org
_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Greg Rundlett (freephile)
I should mention that running runJobs with max memory and a large number of
jobs will slow down your wiki until the jobs queue is complete.  You can
'nice' the script, or use other approaches like putting it into a loop with
delays in between batches to clear your queue.

Greg Rundlett
https://eQuality-Tech.com
https://freephile.org

On Mon, Mar 26, 2018 at 1:46 PM, Greg Rundlett (freephile) <
[hidden email]> wrote:

> Hi Andrew,
>
> As the error says, you're running out of memory on the execution of
> runJobs.php
>
> Try
>
> php maintenance/runJobs.php --memory-limit max --maxjobs 5
>
> If that completes without error, then try increasing the number of jobs
> allowed with --maxjobs
>
> Greg Rundlett
> https://eQuality-Tech.com
> https://freephile.org
>
>
>
_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Andrew Geary
In reply to this post by Greg Rundlett (freephile)
Thank you for the responses Greg. I ran the code and received a slightly
different message:

PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to
allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line 518

It calls a different file for the error. It did not run any job.

Thanks for the troubleshooting.

Andrew
* Listen to SEG’s podcast, Seismic Soundoff *

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA

-----Original Message-----
From: Andrew Geary
Sent: Monday, March 26, 2018 10:43 AM
To: '[hidden email]' <[hidden email]>
Subject: Mediawiki Job queue not running

I administer a public wiki - https://wiki.seg.org - and I am having issues
with a jobs backlog. It's currently at 19,058 and counting. I have tried
using runJobs.php, but I get this error message:

PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to
allocate 3846541 bytes) in
/var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656

I have noticed I do not have the variable:

$wgJobRunRate

listed in my LocalSettings.php.

Should I add this variable to my LocalSettings.php file? How can I address
this error message? I believe the jobs backlog is contributing to a
Translate extension error I am having, mentioned here -
https://phabricator.wikimedia.org/T159947.

I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
No changes have been made to my version of Mediawiki in that timeframe.
Thank you.

Best,
Andrew

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA


_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Brian Wolff
Hmm weird.

So in your original message the memory limit is 150 mb (which is job queue
default). In your second message its 256kb (php default when given an
invalid value), so i think there is something wrong with how setting to max
works.

Try the following
--memory-limit default

If that still doesnt work, try
--memory-limit 512000000

For $wgJobRunRate, if not included in LocalSettings.php, it defaults to 1,
which is probably fine.

--
Brian

On Monday, March 26, 2018, Andrew Geary <[hidden email]> wrote:

> Thank you for the responses Greg. I ran the code and received a slightly
> different message:
>
> PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to
> allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line 518
>
> It calls a different file for the error. It did not run any job.
>
> Thanks for the troubleshooting.
>
> Andrew
> * Listen to SEG’s podcast, Seismic Soundoff *
>
> Andrew Geary | publications outreach editor
> +1.918.497.4615 | [hidden email]
> Society of Exploration Geophysicists (SEG)
> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>
> -----Original Message-----
> From: Andrew Geary
> Sent: Monday, March 26, 2018 10:43 AM
> To: '[hidden email]' <[hidden email]>
> Subject: Mediawiki Job queue not running
>
> I administer a public wiki - https://wiki.seg.org - and I am having issues
> with a jobs backlog. It's currently at 19,058 and counting. I have tried
> using runJobs.php, but I get this error message:
>
> PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried
to

> allocate 3846541 bytes) in
> /var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656
>
> I have noticed I do not have the variable:
>
> $wgJobRunRate
>
> listed in my LocalSettings.php.
>
> Should I add this variable to my LocalSettings.php file? How can I address
> this error message? I believe the jobs backlog is contributing to a
> Translate extension error I am having, mentioned here -
> https://phabricator.wikimedia.org/T159947.
>
> I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
> No changes have been made to my version of Mediawiki in that timeframe.
> Thank you.
>
> Best,
> Andrew
>
> Andrew Geary | publications outreach editor
> +1.918.497.4615 | [hidden email]
> Society of Exploration Geophysicists (SEG)
> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>
>
> _______________________________________________
> MediaWiki-l mailing list
> To unsubscribe, go to:
> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
>
_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Andrew Geary
In reply to this post by Greg Rundlett (freephile)
I appreciate the support Brian and Greg. I am still receiving the same error
message.

PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to
allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line 518

Brian - you clearly set the memory limit above the referenced number, so I'm
not sure the issue. Any additional help would be appreciated. Thank you.

Andrew
* Listen to SEG’s podcast, Seismic Soundoff *

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA

-----Original Message-----
From: Andrew Geary
Sent: Monday, March 26, 2018 10:43 AM
To: '[hidden email]' <[hidden email]>
Subject: Mediawiki Job queue not running

I administer a public wiki - https://wiki.seg.org - and I am having issues
with a jobs backlog. It's currently at 19,058 and counting. I have tried
using runJobs.php, but I get this error message:

PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to
allocate 3846541 bytes) in
/var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656

I have noticed I do not have the variable:

$wgJobRunRate

listed in my LocalSettings.php.

Should I add this variable to my LocalSettings.php file? How can I address
this error message? I believe the jobs backlog is contributing to a
Translate extension error I am having, mentioned here -
https://phabricator.wikimedia.org/T159947.

I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
No changes have been made to my version of Mediawiki in that timeframe.
Thank you.

Best,
Andrew

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA


_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Tom Hutchison
Couple questions for more debugging info...

What is your setup? Distribution running? Memory? If you are running a VM with 512MB of memory, you could easily be running out of memory if the site has high traffic.

What are the older jobs that are stuck? Type of job - aka job_cmd column in the job table.

Run a php -I | grep memory

Are any low values being set and where?

Set $wgJobRunRate = 0;  the default it 1 which means, run a job for every page load. Again, if you have a high traffic volume, a user hitting a page every 1 or 2 secs, your install is trying to run a job every 1 or 2 secs or more.

Caching? What is $wgMainCacheType set to? Are you using an A

Logs, logs, logs... is there any information in the error logs. Have you enabled debugging to see if there are any extensions causing issues?

Tom


-----Original Message-----
From: MediaWiki-l <[hidden email]> On Behalf Of Andrew Geary
Sent: Monday, March 26, 2018 5:01 PM
To: [hidden email]
Subject: Re: [MediaWiki-l] Mediawiki Job queue not running

I appreciate the support Brian and Greg. I am still receiving the same error message.

PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line 518

Brian - you clearly set the memory limit above the referenced number, so I'm not sure the issue. Any additional help would be appreciated. Thank you.

Andrew
* Listen to SEG’s podcast, Seismic Soundoff *

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA

-----Original Message-----
From: Andrew Geary
Sent: Monday, March 26, 2018 10:43 AM
To: '[hidden email]' <[hidden email]>
Subject: Mediawiki Job queue not running

I administer a public wiki - https://wiki.seg.org - and I am having issues with a jobs backlog. It's currently at 19,058 and counting. I have tried using runJobs.php, but I get this error message:

PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to allocate 3846541 bytes) in /var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656

I have noticed I do not have the variable:

$wgJobRunRate

listed in my LocalSettings.php.

Should I add this variable to my LocalSettings.php file? How can I address this error message? I believe the jobs backlog is contributing to a Translate extension error I am having, mentioned here - https://phabricator.wikimedia.org/T159947.

I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
No changes have been made to my version of Mediawiki in that timeframe.
Thank you.

Best,
Andrew

Andrew Geary | publications outreach editor
+1.918.497.4615 | [hidden email]
Society of Exploration Geophysicists (SEG)
8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA


_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l


_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Tom Hutchison
In reply to this post by Andrew Geary
Oops... some clarity

Run a php -i | grep memory (stupid auto capitalization)

And.. are you using APC?

Tom





_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Brian Wolff
In reply to this post by Andrew Geary
I looked into this a little further. It appears the --memory-limit
option for the script is broken (Not only that, its been broken for 8
years). I have a patch to fix it (
https://gerrit.wikimedia.org/r/422359 ) which will hopefully be
included in MediaWiki 1.31.

In the mean time, you can try either:
* Check to see if its just one type of job, and run all the other
types to further narrow down where the problem is. (First run
showJobs.php --group to get a list of job types that you have. And
then run runJobs.php --type PUT_TYPE_NAME_HERE for each of the job
types listed by showJobs.php)
* If you think its just one bad job, you could delete that job (run
sql.php and then do: delete from job order by job_id asc limit 1;),
although I believe job queue would skip the failing job all by itself
if it was just that job.
* Manually upping the memory limit on line 51 of runJobs.php
(Replacing return "150M"; with return "250M"; ) [Or alternatively,
manually applying the gerrit patch I referenced above]

--
Brian


On Mon, Mar 26, 2018 at 9:01 PM, Andrew Geary <[hidden email]> wrote:

> I appreciate the support Brian and Greg. I am still receiving the same error
> message.
>
> PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to
> allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line 518
>
> Brian - you clearly set the memory limit above the referenced number, so I'm
> not sure the issue. Any additional help would be appreciated. Thank you.
>
> Andrew
> * Listen to SEG’s podcast, Seismic Soundoff *
>
> Andrew Geary | publications outreach editor
> +1.918.497.4615 | [hidden email]
> Society of Exploration Geophysicists (SEG)
> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>
> -----Original Message-----
> From: Andrew Geary
> Sent: Monday, March 26, 2018 10:43 AM
> To: '[hidden email]' <[hidden email]>
> Subject: Mediawiki Job queue not running
>
> I administer a public wiki - https://wiki.seg.org - and I am having issues
> with a jobs backlog. It's currently at 19,058 and counting. I have tried
> using runJobs.php, but I get this error message:
>
> PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to
> allocate 3846541 bytes) in
> /var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656
>
> I have noticed I do not have the variable:
>
> $wgJobRunRate
>
> listed in my LocalSettings.php.
>
> Should I add this variable to my LocalSettings.php file? How can I address
> this error message? I believe the jobs backlog is contributing to a
> Translate extension error I am having, mentioned here -
> https://phabricator.wikimedia.org/T159947.
>
> I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
> No changes have been made to my version of Mediawiki in that timeframe.
> Thank you.
>
> Best,
> Andrew
>
> Andrew Geary | publications outreach editor
> +1.918.497.4615 | [hidden email]
> Society of Exploration Geophysicists (SEG)
> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>
>
> _______________________________________________
> MediaWiki-l mailing list
> To unsubscribe, go to:
> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

Brian Wolff
Ok, its been pointed out to me by Niklas that the memory limit option
only works with an equal sign currently.

So you can do:

php runJobs.php --memory-limit=512M

and it should set the memory limit correctly.

--
Brian

On Wed, Mar 28, 2018 at 1:45 AM, Brian Wolff <[hidden email]> wrote:

> I looked into this a little further. It appears the --memory-limit
> option for the script is broken (Not only that, its been broken for 8
> years). I have a patch to fix it (
> https://gerrit.wikimedia.org/r/422359 ) which will hopefully be
> included in MediaWiki 1.31.
>
> In the mean time, you can try either:
> * Check to see if its just one type of job, and run all the other
> types to further narrow down where the problem is. (First run
> showJobs.php --group to get a list of job types that you have. And
> then run runJobs.php --type PUT_TYPE_NAME_HERE for each of the job
> types listed by showJobs.php)
> * If you think its just one bad job, you could delete that job (run
> sql.php and then do: delete from job order by job_id asc limit 1;),
> although I believe job queue would skip the failing job all by itself
> if it was just that job.
> * Manually upping the memory limit on line 51 of runJobs.php
> (Replacing return "150M"; with return "250M"; ) [Or alternatively,
> manually applying the gerrit patch I referenced above]
>
> --
> Brian
>
>
> On Mon, Mar 26, 2018 at 9:01 PM, Andrew Geary <[hidden email]> wrote:
>> I appreciate the support Brian and Greg. I am still receiving the same error
>> message.
>>
>> PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried to
>> allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line 518
>>
>> Brian - you clearly set the memory limit above the referenced number, so I'm
>> not sure the issue. Any additional help would be appreciated. Thank you.
>>
>> Andrew
>> * Listen to SEG’s podcast, Seismic Soundoff *
>>
>> Andrew Geary | publications outreach editor
>> +1.918.497.4615 | [hidden email]
>> Society of Exploration Geophysicists (SEG)
>> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>>
>> -----Original Message-----
>> From: Andrew Geary
>> Sent: Monday, March 26, 2018 10:43 AM
>> To: '[hidden email]' <[hidden email]>
>> Subject: Mediawiki Job queue not running
>>
>> I administer a public wiki - https://wiki.seg.org - and I am having issues
>> with a jobs backlog. It's currently at 19,058 and counting. I have tried
>> using runJobs.php, but I get this error message:
>>
>> PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to
>> allocate 3846541 bytes) in
>> /var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line 656
>>
>> I have noticed I do not have the variable:
>>
>> $wgJobRunRate
>>
>> listed in my LocalSettings.php.
>>
>> Should I add this variable to my LocalSettings.php file? How can I address
>> this error message? I believe the jobs backlog is contributing to a
>> Translate extension error I am having, mentioned here -
>> https://phabricator.wikimedia.org/T159947.
>>
>> I am running MediaWiki 1.26.2. The jobs back log started in November 2017.
>> No changes have been made to my version of Mediawiki in that timeframe.
>> Thank you.
>>
>> Best,
>> Andrew
>>
>> Andrew Geary | publications outreach editor
>> +1.918.497.4615 | [hidden email]
>> Society of Exploration Geophysicists (SEG)
>> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>>
>>
>> _______________________________________________
>> MediaWiki-l mailing list
>> To unsubscribe, go to:
>> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Mediawiki Job queue not running

jaquilina
In reply to this post by Andrew Geary
Hi Sorry to chime in late here, but the memory allocation if im
remembering right is in the php.ini.

Regards,
Jonathan

On 2018-03-26 21:01, Andrew Geary wrote:

> I appreciate the support Brian and Greg. I am still receiving the same
> error
> message.
>
> PHP Fatal error:  Allowed memory size of 262144 bytes exhausted (tried
> to
> allocate 196608 bytes) in /var/www/html/mediawiki/autoload.php on line
> 518
>
> Brian - you clearly set the memory limit above the referenced number,
> so I'm
> not sure the issue. Any additional help would be appreciated. Thank
> you.
>
> Andrew
> * Listen to SEG’s podcast, Seismic Soundoff *
>
> Andrew Geary | publications outreach editor
> +1.918.497.4615 | [hidden email]
> Society of Exploration Geophysicists (SEG)
> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>
> -----Original Message-----
> From: Andrew Geary
> Sent: Monday, March 26, 2018 10:43 AM
> To: '[hidden email]' <[hidden email]>
> Subject: Mediawiki Job queue not running
>
> I administer a public wiki - https://wiki.seg.org - and I am having
> issues
> with a jobs backlog. It's currently at 19,058 and counting. I have
> tried
> using runJobs.php, but I get this error message:
>
> PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted
> (tried to
> allocate 3846541 bytes) in
> /var/www/html/mediawiki/includes/objectcache/SqlBagOStuff.php on line
> 656
>
> I have noticed I do not have the variable:
>
> $wgJobRunRate
>
> listed in my LocalSettings.php.
>
> Should I add this variable to my LocalSettings.php file? How can I
> address
> this error message? I believe the jobs backlog is contributing to a
> Translate extension error I am having, mentioned here -
> https://phabricator.wikimedia.org/T159947.
>
> I am running MediaWiki 1.26.2. The jobs back log started in November
> 2017.
> No changes have been made to my version of Mediawiki in that timeframe.
> Thank you.
>
> Best,
> Andrew
>
> Andrew Geary | publications outreach editor
> +1.918.497.4615 | [hidden email]
> Society of Exploration Geophysicists (SEG)
> 8801 South Yale Ave, Ste. 500, Tulsa, OK 74137 USA
>
>
> _______________________________________________
> MediaWiki-l mailing list
> To unsubscribe, go to:
> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l