Job queue and Redis?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Job queue and Redis?

thingles
I’m pretty sure that WMF is using Redis for the MediaWiki job queue. I see some sparse documentation on setting it up. I already have redis on my machines and would like to move to this to make life easier on my database server, but I’m wondering if this is fully baked and ready for use. Are there folks outside of WMF using Redis for their job queue?

--
Jamie Thingelstad
[hidden email]

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Job queue and Redis?

Dan Andreescu
>
> I'm pretty sure that WMF is using Redis for the MediaWiki job queue. I see
> some sparse documentation on setting it up. I already have redis on my
> machines and would like to move to this to make life easier on my database
> server, but I'm wondering if this is fully baked and ready for use. Are
> there folks outside of WMF using Redis for their job queue?
>

Well, I'm not outside WMF, but with Wikimetrics [1], we use Redis as a
Result Backend for our Celery queue.  It works great and besides me getting
the configuration wrong once, everything is very solid.  I'd be happy to
answer any specific questions if you have them.


[1] https://git.wikimedia.org/summary/analytics%2Fwikimetrics.git
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Job queue and Redis?

Brian Wolff
On Jan 31, 2014 4:52 PM, "Dan Andreescu" <[hidden email]> wrote:
>
> >
> > I'm pretty sure that WMF is using Redis for the MediaWiki job queue. I
see
> > some sparse documentation on setting it up. I already have redis on my
> > machines and would like to move to this to make life easier on my
database
> > server, but I'm wondering if this is fully baked and ready for use. Are
> > there folks outside of WMF using Redis for their job queue?
> >
>
> Well, I'm not outside WMF, but with Wikimetrics [1], we use Redis as a
> Result Backend for our Celery queue.  It works great and besides me
getting
> the configuration wrong once, everything is very solid.  I'd be happy to
> answer any specific questions if you have them.
>
>
> [1] https://git.wikimedia.org/summary/analytics%2Fwikimetrics.git
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

I think he meant if the php code mediawiki uses to interact with redis for
job queue is stable, not redis itself.

--bawolff
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Job queue and Redis?

Chad
In reply to this post by thingles
On Fri, Jan 31, 2014 at 5:21 AM, Jamie Thingelstad <[hidden email]>wrote:

> I’m pretty sure that WMF is using Redis for the MediaWiki job queue. I see
> some sparse documentation on setting it up. I already have redis on my
> machines and would like to move to this to make life easier on my database
> server, but I’m wondering if this is fully baked and ready for use.


Yes your assumptions are correct. I'd say it's fully production ready
now--we use it to run all the job queues at WMF and we're doing
thousands of jobs inserted & processed an hour. If documentation is
a bit lacking, maybe Aaron can be prodded into expanding it :p


> Are there folks outside of WMF using Redis for their job queue?
>
>
Dunno. Maybe Translatewiki? Wikia?

-Chad
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l