[Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Huib Laurens
Hello,

I always believed that our servers where monitored 24/7? But nobody seems
to be arround to fix a core part in our systems?

Huib

---------- Forwarded message ----------
From: rupert THURNER <[hidden email]>
Date: Sat, Aug 10, 2013 at 5:24 PM
Subject: [Wikimania-l] git.wikimedia.org dead due to wikimania ;)
To: "Wikimania general list (open subscription)" <
[hidden email]>


it seems git.wikimedia.org is down due:
* no volunteer has access
* no paid person works on weekends
* everybody else is at wikimania

rupert.

_______________________________________________
Wikimania-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



--
Met vriendelijke groet,

Huib Laurens
_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Martijn Hoekstra
On Sat, Aug 10, 2013 at 5:31 PM, Huib Laurens <[hidden email]> wrote:

> Hello,
>
> I always believed that our servers where monitored 24/7? But nobody seems
> to be arround to fix a core part in our systems?
>
> Huib
>
> ---------- Forwarded message ----------
> From: rupert THURNER <[hidden email]>
> Date: Sat, Aug 10, 2013 at 5:24 PM
> Subject: [Wikimania-l] git.wikimedia.org dead due to wikimania ;)
> To: "Wikimania general list (open subscription)" <
> [hidden email]>
>
>
> it seems git.wikimedia.org is down due:
> * no volunteer has access
> * no paid person works on weekends
> * everybody else is at wikimania
>
> rupert.
>


Doesn't this imply that nobody is at Wikimania?


>
> _______________________________________________
> Wikimania-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>
>
>
> --
> Met vriendelijke groet,
>
> Huib Laurens
> _______________________________________________
> Wikimedia-l mailing list
> [hidden email]
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:[hidden email]?subject=unsubscribe>
_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Federico Leva (Nemo)
In reply to this post by Huib Laurens
Huib Laurens, 10/08/2013 17:31:
> I always believed that our servers where monitored 24/7?

Not magically, despite the amount of unicorns bought by the WMF. You're
looking for https://bugzilla.wikimedia.org/show_bug.cgi?id=51983

> But nobody seems
> to be arround to fix a core part in our systems?

No development is really prevented by gitblit being down; business as
usual continues on gerrit.wikimedia.org and this additional reliability
is actually what they wanted to achieve by moving gitblit (formerly
gitweb) to a separate server.

Nemo

_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

MZMcBride-2
In reply to this post by Huib Laurens
Huib Laurens wrote:
>I always believed that our servers were monitored 24/7? But nobody seems
>to be around to fix a core part in our systems?

Hi Huib.

You've been around quite a long time, so it shouldn't be new information
to you that the appropriate mailing list for an issue like this is
wikitech-l (where there's already an ongoing thread), not wikimedia-l. And
you should also know that for issues like this, the best place to search
is Bugzilla, as both Nemo and myself have now pointed out (specifically
<https://bugzilla.wikimedia.org/51769>).

You seem to be a bit confused about core v. non-core services. In the
Wikimedia world, core functionality generally means that the wikis are
accessible for, at the very minimum, read access. Peripheral services and
sites have varying levels of criticality, though there's no rating system
in which git.wikimedia.org (a simple Git repository viewer) would be
considered core (though it being completely down can be considered
critical, as it makes development work more tedious and annoying).

Obviously the operations team has a number of monitoring systems in place.
The public monitoring dashboard is here: <http://status.wikimedia.org>.

MZMcBride



_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Rich Farmbrough
In reply to this post by Huib Laurens
I maintained 24/7 support with a team of 6. WMF has 150 staff and does
not have weekend support.  The tail is wagging the dog.


_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Oliver Keyes-4
We have weekend support - for core services, which constitute our MediaWiki
instances. Git, however, is not a core service - as Max accurately notes,
while it makes development finicky and frustrating, it doesn't impact on
our core mission, which is to provide the sum of human knowledge, for
everybody. It being down for a short period of time is not critical -
Wikipedia being down? Well, I think we'd agree that's a pretty serious
problem.

I'd like to think Engineering do a pretty good job at uptime for core
services - when was the last time you saw Wikipedia down for any extended
period of time? - regardless of what day of the week it is. I recall a few
months ago that we had weird, inconsistent cacheing problems that were
causing trouble for users - Leslie Carr, one of our Ops engineers, worked
11 hours on Friday and (iirc) 14 hours over the weekend to get it fixed.
This isn't the only story in that vein - where there are critical problems
to critical services, they tend to be fixed as promptly as possible, and
staffers pull out all of the stops to make it happen. But it's not
realistic for ops and dev/ops to maintain 24/7 uptime not just for our core
services, but for every single one of the miscellaneous instances we host.
If we had 150 engineers in Platform and Operations, then I might be singing
a different tune, but unfortunately we don't ;).

On the other hand, if you would like to donate enough money to pay for a
team of 6 to maintain our git instance 24/7, I'm sure we could set
something up :P


On 12 August 2013 20:19, Richard Farmbrough <[hidden email]>wrote:

> I maintained 24/7 support with a team of 6. WMF has 150 staff and does not
> have weekend support.  The tail is wagging the dog.
>
>
>
> ______________________________**_________________
> Wikimedia-l mailing list
> [hidden email].**org <[hidden email]>
> Unsubscribe: https://lists.wikimedia.org/**mailman/listinfo/wikimedia-l<https://lists.wikimedia.org/mailman/listinfo/wikimedia-l>,
> <mailto:wikimedia-l-request@**lists.wikimedia.org<[hidden email]>
> ?subject=**unsubscribe>
>



--
Oliver Keyes
Community Liaison, Product Development
Wikimedia Foundation
_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Greg Grossmeier-2
A quick update I just sent to wikitech-l (where this conversation should
be happening):

http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071171.html

Full text below:

----------------

As you all are aware, git.wikimedia.org (powered by GitBlit) has been
unstable. The root cause is basically that there is insufficient caching
mechanisms in place exacerbated by a lack of robust rules for web
indexers (Google, Bing, etc).

Currently, the solution is to disallow all indexers from indexing
git.wikimedia.org via the robots.txt file
(https://git.wikimedia.org/robots.txt). We hope to open up indexing in a
more controlled fashion soon.

For now, git.wikimedia.org should be stable for everyday use.

You can follow along with the caching developments on bug 49371:
https://bugzilla.wikimedia.org/show_bug.cgi?id=49371

Thanks to Chad and Ariel, especially, who have been working to make sure
this is resolved quickly and permanently.

Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>

signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Chris McKenna
In reply to this post by Oliver Keyes-4
On Tue, 13 Aug 2013, Oliver Keyes wrote:
>
> I'd like to think Engineering do a pretty good job at uptime for core
> services - when was the last time you saw Wikipedia down for any extended
> period of time? - regardless of what day of the week it is.

Wikipedia uptime is certianly much better these days than it was around
2005/6 when there were at least two websites dedicated to reporting its
status. Back then it wasn't uncommon to experience major slowdowns,
periods of non-responsiveness, extended read only periods and edits
failing (sometimes silently) due to (iirc) overloaded servers. I can't
remember the last time I saw anything like that, but it certainly wasn't
2013.

The folks responsible for this certainly deserve kudos for their efforts.

----
Chris McKenna

[hidden email]
www.sucs.org/~cmckenna


The essential things in life are seen not with the eyes,
but with the heart

Antoine de Saint Exupery


_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Tim Starling-2
In reply to this post by Oliver Keyes-4
On 14/08/13 07:40, Oliver Keyes wrote:
> We have weekend support - for core services, which constitute our MediaWiki
> instances. Git, however, is not a core service - as Max accurately notes,
> while it makes development finicky and frustrating,

I didn't know anyone used git.wikimedia.org. I think I've only visited
it once.

We're not talking about Git being down, just about one of the two web
viewers being down (the other being GitHub). Web viewers of source
code are a luxury -- they're handy when you're not a developer and so
couldn't be bothered to clone the repository, but they're hardly a
human right.

Despite that, note that Leslie Carr did work on fixing it on the weekend.

-- Tim Starling


_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Amir E. Aharoni
2013/8/14 Tim Starling <[hidden email]>:
> On 14/08/13 07:40, Oliver Keyes wrote:
>> We have weekend support - for core services, which constitute our MediaWiki
>> instances. Git, however, is not a core service - as Max accurately notes,
>> while it makes development finicky and frustrating,
>
> I didn't know anyone used git.wikimedia.org. I think I've only visited
> it once.

I sometimes use it to point people to code, especially new volunteers
who are starting to learn the code.

Other people very frequently send me git.wikimedia.org links when they
are asking me about the code.

So yes, it's not as important as keeping *.wikipedia.org online, but
it is definitely far from negligible.

--
Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
http://aharoni.wordpress.com
‪“We're living in pieces,
I want to live in peace.” – T. Moore‬

_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Federico Leva (Nemo)
In reply to this post by Chris McKenna
Chris McKenna, 14/08/2013 00:00:

> On Tue, 13 Aug 2013, Oliver Keyes wrote:
>>
>> I'd like to think Engineering do a pretty good job at uptime for core
>> services - when was the last time you saw Wikipedia down for any extended
>> period of time? - regardless of what day of the week it is.
>
> Wikipedia uptime is certianly much better these days than it was around
> 2005/6 when there were at least two websites dedicated to reporting its
> status. Back then it wasn't uncommon to experience major slowdowns,
> periods of non-responsiveness, extended read only periods and edits
> failing (sometimes silently) due to (iirc) overloaded servers. I can't
> remember the last time I saw anything like that, but it certainly wasn't
> 2013.

Btw it would be nice to have stats on uptime, or at least some
assessment of the progress for the "stabilizing infrastructure" 5 years
strategic goal. I couldn't find any recently, I asked both on mailing
lists and on Meta.
<https://meta.wikimedia.org/wiki/Talk:Wikimedia_budget#Stabilizing_the_infrastructure>

Nemo

_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Leslie Carr
http://status.wikimedia.org is from external monitors

Sent from my mobile. Please excuse the brevity and typos.
On Aug 14, 2013 4:44 AM, "Federico Leva (Nemo)" <[hidden email]> wrote:

> Chris McKenna, 14/08/2013 00:00:
>
>> On Tue, 13 Aug 2013, Oliver Keyes wrote:
>>
>>>
>>> I'd like to think Engineering do a pretty good job at uptime for core
>>> services - when was the last time you saw Wikipedia down for any extended
>>> period of time? - regardless of what day of the week it is.
>>>
>>
>> Wikipedia uptime is certianly much better these days than it was around
>> 2005/6 when there were at least two websites dedicated to reporting its
>> status. Back then it wasn't uncommon to experience major slowdowns,
>> periods of non-responsiveness, extended read only periods and edits
>> failing (sometimes silently) due to (iirc) overloaded servers. I can't
>> remember the last time I saw anything like that, but it certainly wasn't
>> 2013.
>>
>
> Btw it would be nice to have stats on uptime, or at least some assessment
> of the progress for the "stabilizing infrastructure" 5 years strategic
> goal. I couldn't find any recently, I asked both on mailing lists and on
> Meta.
> <https://meta.wikimedia.org/**wiki/Talk:Wikimedia_budget#**
> Stabilizing_the_infrastructure<https://meta.wikimedia.org/wiki/Talk:Wikimedia_budget#Stabilizing_the_infrastructure>
> **>
>
> Nemo
>
> ______________________________**_________________
> Wikimedia-l mailing list
> [hidden email].**org <[hidden email]>
> Unsubscribe: https://lists.wikimedia.org/**mailman/listinfo/wikimedia-l<https://lists.wikimedia.org/mailman/listinfo/wikimedia-l>,
> <mailto:wikimedia-l-request@**lists.wikimedia.org<[hidden email]>
> ?subject=**unsubscribe>
_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>
Reply | Threaded
Open this post in threaded view
|

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

Federico Leva (Nemo)
Leslie Carr, 14/08/2013 22:30:
> http://status.wikimedia.org is from external monitors

I don't see any annual or multi-year stats there and I don't think they
monitor readon-only vs. rw state.

Nemo

_______________________________________________
Wikimedia-l mailing list
[hidden email]
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[hidden email]?subject=unsubscribe>