Could not acquire (database)_message:en:status lock

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Could not acquire (database)_message:en:status lock

Justin Folvarcik
I've been having this problem with my production setup, and we are
currently down because I can't seem to rectify it. Everything points to a
cache issue, for which we use redis. I've tried restarting our redis
server, as well as our MariaDB install and even nginx on both web servers.
So far, I have yet to make the site usable. Can anyone at least point me in
the right direction of how to start debugging? The site was working just
fine and was actually running pretty quickly up until a few hours ago, when
this error suddenly appeared and took down the site.

This is with a load-balanced server setup including two web servers with
nginx, a PHP-FPM server, a MariaDB server, a redis server, and an NFS file
server which contains the codebase. All of them run on Ubuntu 14.04.

Any help is greatly appreciated.
----
Justin Folvarcik
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Could not acquire (database)_message:en:status lock

Jeremy Baron
Hi,

On Mar 7, 2015 9:56 PM, "Justin Folvarcik" <[hidden email]> wrote:
> Can anyone at least point me in
> the right direction of how to start debugging?

https://phabricator.wikimedia.org/T46499
https://phabricator.wikimedia.org/T45516
https://phabricator.wikimedia.org/T44127

Maybe your redis filled up? you should try emptying the cache.

(in general:)

https://www.mediawiki.org/wiki/Manual:How_to_debug

you could turn up the log level, turn on extra logs, figure out what part
of the codebase is the immediate source of the message (e.g. by changing
messages in the source and checking if the site has a corresponding change
in message), print a stack trace and local variable state whenever that
code path is hit

or use a real debugger :)

-Jeremy
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l