Possible LDAP outtime this morning, major disruption

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Possible LDAP outtime this morning, major disruption

Tim Landscheidt
Hi,

from about 3:00Z to about 3:20Z, no login was possible to
nightshade and yarrow, (not existing) passwords required for
willow and the webserver returned 404s.  MZMcBride had an
open session into willow, and loads of accessible servers
were within limits
(cf. http://p.defau.lt/?e_zsJIW_rAbfR3Cvlvx9Uw), but reserve
lookup of user names was broken
(cf. http://p.defau.lt/?asmBijtXnvzQacz1e8JXOQ) and
ldapsearch timed out as well
(cf. http://p.defau.lt/?P47PCC3_1d3mnoLyVqFUqQ).  This looks
like a failure of the LDAP server.

Two other issues surfaced at that time:

- http://nagios.toolserver.org/ gave 500s during the outage.
  I asked Coren to consult with WMF if there are possibili-
  ties to outsource (or integrate :-)) this monitoring to
  their existing infrastructure
  (http://icinga.wikimedia.org/).

- The listed mail address for the Toolserver admins is
  [hidden email].  While this may work during such
  an outage (I didn't try) and personal mail addresses for
  admins can be found in the toolserver-announce archives,
  we should prefer an address routed externally, and trying
  not to be too imaginative I propose:
  [hidden email].

Tim


_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Possible LDAP outtime this morning, major disruption

Tim Landscheidt
I wrote:

> [...]

> - The listed mail address for the Toolserver admins is
>   [hidden email].  While this may work during such
>   an outage (I didn't try) and personal mail addresses for
>   admins can be found in the toolserver-announce archives,
>   we should prefer an address routed externally, and trying
>   not to be too imaginative I propose:
>   [hidden email].

And imagine how unimaginative I can be, as according to Na-
gios this address seems to exist already :-).

Tim


_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Possible LDAP outtime this morning, major disruption

Federico Leva (Nemo)
Just tested because same LDAP problems are happening right now:
<[hidden email]>: 554 Sorry, no mailbox here by that name.
But toolserver.org works.

Nemo

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Reply | Threaded
Open this post in threaded view
|

Re: Possible LDAP outtime this morning, major disruption

John Doe-27
Looks like this is causing a serious TS wide outage

On Fri, Mar 1, 2013 at 7:39 AM, Federico Leva (Nemo) <[hidden email]> wrote:

> Just tested because same LDAP problems are happening right now:
> <[hidden email]>: 554 Sorry, no mailbox here by that name.
> But toolserver.org works.
>
> Nemo
>
>
> _______________________________________________
> Toolserver-l mailing list ([hidden email])
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette

_______________________________________________
Toolserver-l mailing list ([hidden email])
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette