Postmortem: General downtime yesterday

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Postmortem: General downtime yesterday

Hello all,

during the maintenance window yesterday evening the hole cluster was down for
~30min starting ~21:20 UTC. The problem was independent of the maintenance
working, but caused the window to extend.
The problem was an out-of-memory on one of our HA-nodes. Unfortunately the box
did not restart itself and its ha-buddy did not detect the problem too, so the
services of the out-of-memory-box were not switched to the other box. This
caused the hole cluster to stand until I manually rebooted the host. I will
look if I can find some kind of sensor for that; in worst case I will enable
our old "reboot if low on memory"-script again.


Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885

Toolserver-l mailing list ([hidden email])
Posting guidelines for this list:

signature.asc (205 bytes) Download Attachment