Quantcast

[Wikimedia-l] Widespread perfomance issues

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Wikimedia-l] Widespread perfomance issues

Cristian Consonni-3
Hi,

At Wikimedia Italia have been contacted in the last few minutes because
Wikipedia seems unreachable and very slow from Italy.

The status pages for servers and services signals several perfomance issues:
https://status.wikimedia.org/

Does somebody know what's going on?

Thank you.

Cristian

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Cristian Consonni-3
On 15/05/2017 11:02, Cristian Consonni wrote:
> Hi,
>
> At Wikimedia Italia have been contacted in the last few minutes because
> Wikipedia seems unreachable and very slow from Italy.

(Ah, I forgot to mention. by a journalist from ANSA, the national news
agency)

Cristian

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Oloruntoba Oyeyele
Throughout last night (around 7pm WAT till I slept off), I couldn't access
Wikipedia in Nigeria. I thought it was my service provider acting funny at
first. I'll like to know what the problem is, as well.

Sam.

On Mon, May 15, 2017 at 10:03 AM, Cristian Consonni <[hidden email]>
wrote:

> On 15/05/2017 11:02, Cristian Consonni wrote:
> > Hi,
> >
> > At Wikimedia Italia have been contacted in the last few minutes because
> > Wikipedia seems unreachable and very slow from Italy.
>
> (Ah, I forgot to mention. by a journalist from ANSA, the national news
> agency)
>
> Cristian
>
> _______________________________________________
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> New messages to: [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:[hidden email]?subject=unsubscribe>
>
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

kayode yussuf
Hi Sam,
if this will help, I think it was your provider. I was able to access Wikipedia at that same time last night. Wikipedia is also up as at this morning. Kayode Yussuf

    On Monday, May 15, 2017 10:40 AM, Oloruntoba Oyeyele <[hidden email]> wrote:
 

 Throughout last night (around 7pm WAT till I slept off), I couldn't access
Wikipedia in Nigeria. I thought it was my service provider acting funny at
first. I'll like to know what the problem is, as well.

Sam.

On Mon, May 15, 2017 at 10:03 AM, Cristian Consonni <[hidden email]>
wrote:

> On 15/05/2017 11:02, Cristian Consonni wrote:
> > Hi,
> >
> > At Wikimedia Italia have been contacted in the last few minutes because
> > Wikipedia seems unreachable and very slow from Italy.
>
> (Ah, I forgot to mention. by a journalist from ANSA, the national news
> agency)
>
> Cristian
>
> _______________________________________________
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> New messages to: [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:[hidden email]?subject=unsubscribe>
>
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>

   
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Faidon Liambotis
In reply to this post by Cristian Consonni-3
Hi Cristian,

We've had some network connectivity issues with one of our ISPs in our
European datacenter that were probably the source of your problems
(FastWeb in Italy were among the affected ISPs). This was a localized
issue -- only in the European datacenter and one out of six major
network carriers and dozens of network connectivity partners.

The issue has been worked around as of 09:26 UTC and the situation is
being monitored.

This issue is being tracked at:
https://phabricator.wikimedia.org/T165288

Do let us know if this is not the case for you of if you're experiencing
any trouble, here, off-list or directly on that Phabricator task.

Thanks,
Faidon
--
Faidon Liambotis
Principal Operations Engineer
Wikimedia Foundation

On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:

> Hi,
>
> At Wikimedia Italia have been contacted in the last few minutes because
> Wikipedia seems unreachable and very slow from Italy.
>
> The status pages for servers and services signals several perfomance issues:
> https://status.wikimedia.org/
>
> Does somebody know what's going on?
>
> Thank you.
>
> Cristian
>
> _______________________________________________
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Gergő Tisza
In reply to this post by Cristian Consonni-3
On Mon, May 15, 2017 at 11:02 AM, Cristian Consonni <[hidden email]>
wrote:

> At Wikimedia Italia have been contacted in the last few minutes because
> Wikipedia seems unreachable and very slow from Italy.
>
> The status pages for servers and services signals several perfomance
> issues:
> https://status.wikimedia.org/
>
> Does somebody know what's going on?
>

Maybe related to https://phabricator.wikimedia.org/T165252 (hardware
problems in one of the caching servers)?
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Alexandros Kosiaris
Hi,

Probably https://phabricator.wikimedia.org/T165288.

On Mon, May 15, 2017 at 2:04 PM, Gergő Tisza <[hidden email]> wrote:

> On Mon, May 15, 2017 at 11:02 AM, Cristian Consonni <[hidden email]>
> wrote:
>
>> At Wikimedia Italia have been contacted in the last few minutes because
>> Wikipedia seems unreachable and very slow from Italy.
>>
>> The status pages for servers and services signals several perfomance
>> issues:
>> https://status.wikimedia.org/
>>
>> Does somebody know what's going on?
>>
>
> Maybe related to https://phabricator.wikimedia.org/T165252 (hardware
> problems in one of the caching servers)?
> _______________________________________________
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>



--
Alexandros Kosiaris <[hidden email]>

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Cristian Consonni-3
In reply to this post by Faidon Liambotis
Hi Faidon,

On 15/05/2017 13:02, Faidon Liambotis wrote:

> Hi Cristian,
>
> We've had some network connectivity issues with one of our ISPs in our
> European datacenter that were probably the source of your problems
> (FastWeb in Italy were among the affected ISPs). This was a localized
> issue -- only in the European datacenter and one out of six major
> network carriers and dozens of network connectivity partners.
>
> The issue has been worked around as of 09:26 UTC and the situation is
> being monitored.
>
> This issue is being tracked at:
> https://phabricator.wikimedia.org/T165288
>
> Do let us know if this is not the case for you of if you're experiencing
> any trouble, here, off-list or directly on that Phabricator task.

Thanks for the update.
Keep up the good work.

Ciao,

Cristian

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Rogol Domedonfors
In reply to this post by Faidon Liambotis
Faidon

Do you believe that your recovery plan was adequate or will you be
reviewing it in the light of this widespread outage?

"Rogol"

On Mon, May 15, 2017 at 12:02 PM, Faidon Liambotis <[hidden email]>
wrote:

> Hi Cristian,
>
> We've had some network connectivity issues with one of our ISPs in our
> European datacenter that were probably the source of your problems
> (FastWeb in Italy were among the affected ISPs). This was a localized
> issue -- only in the European datacenter and one out of six major
> network carriers and dozens of network connectivity partners.
>
> The issue has been worked around as of 09:26 UTC and the situation is
> being monitored.
>
> This issue is being tracked at:
> https://phabricator.wikimedia.org/T165288
>
> Do let us know if this is not the case for you of if you're experiencing
> any trouble, here, off-list or directly on that Phabricator task.
>
> Thanks,
> Faidon
> --
> Faidon Liambotis
> Principal Operations Engineer
> Wikimedia Foundation
>
> On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
> > Hi,
> >
> > At Wikimedia Italia have been contacted in the last few minutes because
> > Wikipedia seems unreachable and very slow from Italy.
> >
> > The status pages for servers and services signals several perfomance
> issues:
> > https://status.wikimedia.org/
> >
> > Does somebody know what's going on?
> >
> > Thank you.
> >
> > Cristian
> >
> > _______________________________________________
> > Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> > New messages to: [hidden email]
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:[hidden email]?subject=unsubscribe>
>
> _______________________________________________
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> New messages to: [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:[hidden email]?subject=unsubscribe>
>
_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Faidon Liambotis
Hi Rogol,

I'm not sure what you're referring to when you're talking about a
"recovery plan". These are all pretty standard day-to-day operations for
us.

This was a small outage, affecting only a small portion of European
traffic (and tiny portion of our global traffic) and not what we'd call
a "widespread" one -- very far from it. It wasn't even noticeable in
traffic graphs at all. This was the result of a networking issue in an
intermediate provider between us and the ISPs affected, so nothing that
was in our direct control.

This could have been remediated by either a) the intermediate provider
fixing or working around the issue b) us disabling the use of that
provider temporarily c) the affected ISPs (like Fastweb) disabling the
use of that provider temporarily. In this case, we did not wait for (a)
or (c) and worked around the issue ourselves, faster than Fastweb etc.
did (and in less than an hour even since the initial report). I think
the response from our side and our engineers (and especially Arzhel) was
pretty stellar.

Faidon

On Mon, May 15, 2017 at 09:00:06PM +0100, Rogol Domedonfors wrote:

> Faidon
>
> Do you believe that your recovery plan was adequate or will you be
> reviewing it in the light of this widespread outage?
>
> "Rogol"
>
> On Mon, May 15, 2017 at 12:02 PM, Faidon Liambotis <[hidden email]>
> wrote:
>
> > Hi Cristian,
> >
> > We've had some network connectivity issues with one of our ISPs in our
> > European datacenter that were probably the source of your problems
> > (FastWeb in Italy were among the affected ISPs). This was a localized
> > issue -- only in the European datacenter and one out of six major
> > network carriers and dozens of network connectivity partners.
> >
> > The issue has been worked around as of 09:26 UTC and the situation is
> > being monitored.
> >
> > This issue is being tracked at:
> > https://phabricator.wikimedia.org/T165288
> >
> > Do let us know if this is not the case for you of if you're experiencing
> > any trouble, here, off-list or directly on that Phabricator task.
> >
> > Thanks,
> > Faidon
> > --
> > Faidon Liambotis
> > Principal Operations Engineer
> > Wikimedia Foundation
> >
> > On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
> > > Hi,
> > >
> > > At Wikimedia Italia have been contacted in the last few minutes because
> > > Wikipedia seems unreachable and very slow from Italy.
> > >
> > > The status pages for servers and services signals several perfomance
> > issues:
> > > https://status.wikimedia.org/
> > >
> > > Does somebody know what's going on?
> > >
> > > Thank you.
> > >
> > > Cristian
> > >
> > > _______________________________________________
> > > Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> > wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> > wiki/Wikimedia-l
> > > New messages to: [hidden email]
> > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> > <mailto:[hidden email]?subject=unsubscribe>
> >
> > _______________________________________________
> > Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> > wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> > wiki/Wikimedia-l
> > New messages to: [hidden email]
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> > <mailto:[hidden email]?subject=unsubscribe>
> >
> _______________________________________________
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Wikimedia-l] Widespread perfomance issues

Cristian Consonni-3
Hi Rogol,

On 16/05/2017 15:44, Faidon Liambotis wrote:
> This was a small outage, affecting only a small portion of European
> traffic (and tiny portion of our global traffic) and not what we'd call
> a "widespread" one -- very far from it.

+1
I think I have to apologize to the Operations team for the the wording I
have chosen for the title of this thread. In hindsight, this issue was
minor.

I was taken aback by the combination of the fact that we (at WM-IT) were
contacted by a journalist (which, in hindsight, put on us some pressure
about trying to understand the causes of the problem); we were in fact
experiencing some problems (both Wikimedia Italia's office and myself at
home use Fastweb as ISP); and, finally, I have misjudged the information
contained in the status page.

> This could have been remediated by either a) the intermediate provider
> fixing or working around the issue b) us disabling the use of that
> provider temporarily c) the affected ISPs (like Fastweb) disabling the
> use of that provider temporarily. In this case, we did not wait for (a)
> or (c) and worked around the issue ourselves, faster than Fastweb etc.
> did (and in less than an hour even since the initial report). I think
> the response from our side and our engineers (and especially Arzhel) was
> pretty stellar.

It was.
Thank you, Faidon, and thanks to all the team for the amazing work you do.

Cristian

_______________________________________________
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: [hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Loading...