Gerrit outage

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Gerrit outage

John Bennett
Hello,


On 16 March 2019, Wikimedia Foundation staff observed suspicious activity
associated with Gerrit and as a precautionary step has taken Gerrit offline
pending investigation.


The Wikimedia Foundation's Security, Site Reliability Engineering and
Release Engineering teams are investigating this incident as well as
potential improvements to prevent future incidents. More information will
be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it
becomes available and is confirmed. If you have any questions, please
contact the Security ([hidden email] <[hidden email]>
).


Thanks
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

John Bennett
Hello,

Gerrit is available again but we are continuing to investigate the
suspicious activity.  Our preliminary findings point to no users or
production systems being compromised and no loss of any confidential
information. As we continue to investigate over the next few days we will
add any appropriate updates to the phabricator task (
https://phabricator.wikimedia.org/T218472 ) .

Thanks


On Sat, Mar 16, 2019 at 10:26 AM John Bennett <[hidden email]>
wrote:

> Hello,
>
>
> On 16 March 2019, Wikimedia Foundation staff observed suspicious activity
> associated with Gerrit and as a precautionary step has taken Gerrit offline
> pending investigation.
>
>
> The Wikimedia Foundation's Security, Site Reliability Engineering and
> Release Engineering teams are investigating this incident as well as
> potential improvements to prevent future incidents. More information will
> be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as
> it becomes available and is confirmed. If you have any questions, please
> contact the Security ([hidden email]
> <[hidden email]>).
>
>
> Thanks
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Pine W
Thanks for the updates and for everyone who was or is working on a weekend
day. Sometime in the next few weeks if you can publish an incident report
that has any sensitive information redacted, I would like to read it.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )




On Sat, Mar 16, 2019, 12:25 PM John Bennett <[hidden email]> wrote:

> Hello,
>
> Gerrit is available again but we are continuing to investigate the
> suspicious activity.  Our preliminary findings point to no users or
> production systems being compromised and no loss of any confidential
> information. As we continue to investigate over the next few days we will
> add any appropriate updates to the phabricator task (
> https://phabricator.wikimedia.org/T218472 ) .
>
> Thanks
>
>
> On Sat, Mar 16, 2019 at 10:26 AM John Bennett <[hidden email]>
> wrote:
>
> > Hello,
> >
> >
> > On 16 March 2019, Wikimedia Foundation staff observed suspicious activity
> > associated with Gerrit and as a precautionary step has taken Gerrit
> offline
> > pending investigation.
> >
> >
> > The Wikimedia Foundation's Security, Site Reliability Engineering and
> > Release Engineering teams are investigating this incident as well as
> > potential improvements to prevent future incidents. More information will
> > be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as
> > it becomes available and is confirmed. If you have any questions, please
> > contact the Security ([hidden email]
> > <[hidden email]>).
> >
> >
> > Thanks
> >
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

MGChecker
Hello,

after the Gerrit outage, my list of watched changes is seemingly empty, while it has not been a day before. Is it possible to fix this issue?

MGChecker

-----Ursprüngliche Nachricht-----
Von: Wikitech-l [mailto:[hidden email]] Im Auftrag von Pine W
Gesendet: Samstag, 16. März 2019 21:14
An: Wikimedia developers
Betreff: Re: [Wikitech-l] Gerrit outage

Thanks for the updates and for everyone who was or is working on a weekend day. Sometime in the next few weeks if you can publish an incident report that has any sensitive information redacted, I would like to read it.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )




On Sat, Mar 16, 2019, 12:25 PM John Bennett <[hidden email]> wrote:

> Hello,
>
> Gerrit is available again but we are continuing to investigate the
> suspicious activity.  Our preliminary findings point to no users or
> production systems being compromised and no loss of any confidential
> information. As we continue to investigate over the next few days we
> will add any appropriate updates to the phabricator task (
> https://phabricator.wikimedia.org/T218472 ) .
>
> Thanks
>
>
> On Sat, Mar 16, 2019 at 10:26 AM John Bennett <[hidden email]>
> wrote:
>
> > Hello,
> >
> >
> > On 16 March 2019, Wikimedia Foundation staff observed suspicious
> > activity associated with Gerrit and as a precautionary step has
> > taken Gerrit
> offline
> > pending investigation.
> >
> >
> > The Wikimedia Foundation's Security, Site Reliability Engineering
> > and Release Engineering teams are investigating this incident as
> > well as potential improvements to prevent future incidents. More
> > information will be posted on Phabricator
> > (https://phabricator.wikimedia.org/T218472 ) as it becomes available
> > and is confirmed. If you have any questions, please contact the
> > Security ([hidden email] <[hidden email]>).
> >
> >
> > Thanks
> >
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Wikipedia Developers mailing list
 The watchlist should be in All-Users. If someone removed your watchlist you can clone that repo. Then in .git/config change refs/heads to just refs/*. Then it should show some refs (I forget which one you check out so you should try them all) you can use git log to see if someone git committed the removal.  
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

John Bennett
In reply to this post by John Bennett
Hello,

Today we have seen Phabricator vandalism from an attacker who was also
responsible for the Gerrit outage yesterday. I’d like to clarify a comment
I made yesterday and provide as many additional details as I can while
still maintaining operational security.

While no user accounts were compromised the attacker leveraged a
vulnerability in Gerrit to comprise a single staff account. This discovery
is what lead to taking Gerrit offline so an investigation could occur, the
vulnerability could be remediated and the service restored.  However, no
further evidence of compromise was discovered and additional security
controls prevented malicious activities from being executed using the
compromised staff account. We will continue to monitor the situation and
will provide updates on this list and on the Phabricator task
https://phabricator.wikimedia.org/T218472.

Thanks

John


On Sat, Mar 16, 2019 at 2:25 PM John Bennett <[hidden email]> wrote:

> Hello,
>
> Gerrit is available again but we are continuing to investigate the
> suspicious activity.  Our preliminary findings point to no users or
> production systems being compromised and no loss of any confidential
> information. As we continue to investigate over the next few days we will
> add any appropriate updates to the phabricator task (
> https://phabricator.wikimedia.org/T218472 ) .
>
> Thanks
>
>
> On Sat, Mar 16, 2019 at 10:26 AM John Bennett <[hidden email]>
> wrote:
>
>> Hello,
>>
>>
>> On 16 March 2019, Wikimedia Foundation staff observed suspicious activity
>> associated with Gerrit and as a precautionary step has taken Gerrit offline
>> pending investigation.
>>
>>
>> The Wikimedia Foundation's Security, Site Reliability Engineering and
>> Release Engineering teams are investigating this incident as well as
>> potential improvements to prevent future incidents. More information will
>> be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as
>> it becomes available and is confirmed. If you have any questions, please
>> contact the Security ([hidden email]
>> <[hidden email]>).
>>
>>
>> Thanks
>>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Tyler Cipriani
In reply to this post by John Bennett
Hello,

As part of cleanup and response Gerrit's use of http tokens has been
disabled. You should still be able to use the http REST api using your
LDAP password.

Gerrit's command-line tools [0] that operate via SSH are also still
available.

-- Tyler

[0]. <https://gerrit.wikimedia.org/r/Documentation/cmd-index.html>

On 19-03-16 10:26:52, John Bennett wrote:

>Hello,
>
>
>On 16 March 2019, Wikimedia Foundation staff observed suspicious activity
>associated with Gerrit and as a precautionary step has taken Gerrit offline
>pending investigation.
>
>
>The Wikimedia Foundation's Security, Site Reliability Engineering and
>Release Engineering teams are investigating this incident as well as
>potential improvements to prevent future incidents. More information will
>be posted on Phabricator (https://phabricator.wikimedia.org/T218472 ) as it
>becomes available and is confirmed. If you have any questions, please
>contact the Security ([hidden email] <[hidden email]>
>).
>
>
>Thanks
>_______________________________________________
>Wikitech-l mailing list
>[hidden email]
>https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

planetenxin
In reply to this post by John Bennett
Gerrit seems to be offline again.

> On 16 March 2019, Wikimedia Foundation staff observed suspicious activity
> associated with Gerrit and as a precautionary step has taken Gerrit offline
> pending investigation.

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Andre Klapper-2
On Tue, 2019-03-19 at 10:59 +0100, planetenxin wrote:
> Gerrit seems to be offline again.

Please read the other latest thread on this very mailing list.

andre
--
Andre Klapper | Bugwrangler / Developer Advocate
https://blogs.gnome.org/aklapper/



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Wikipedia Developers mailing list
Not everyone is aware that the process of cleaning up the vandalism/fixing
Gerrit includes Gerrit being down temporarily.

Do I need to include a reminder link to WP:AGF / WP:DICK?

-- Lewis Cawte

On Tue, 19 Mar 2019, 10:27 Andre Klapper, <[hidden email]> wrote:

> On Tue, 2019-03-19 at 10:59 +0100, planetenxin wrote:
> > Gerrit seems to be offline again.
>
> Please read the other latest thread on this very mailing list.
>
> andre
> --
> Andre Klapper | Bugwrangler / Developer Advocate
> https://blogs.gnome.org/aklapper/
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Dan Garry (Deskana)
On Tue, 19 Mar 2019 at 10:50, Lewis Cawte via Wikitech-l <
[hidden email]> wrote:

> Not everyone is aware that the process of cleaning up the vandalism/fixing
> Gerrit includes Gerrit being down temporarily.
>
> Do I need to include a reminder link to WP:AGF / WP:DICK?
>

That would not be helpful. By assuming that the other person simply missed
the other thread on this mailing list, and by pointing the person to said
thread, Andre *is* assuming good faith.

Dan
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Andre Klapper-2
In reply to this post by Wikipedia Developers mailing list
On Tue, 2019-03-19 at 10:49 +0000, Lewis Cawte via Wikitech-l wrote:
> Not everyone is aware that the process of cleaning up the
> vandalism/fixing Gerrit includes Gerrit being down temporarily.

Right. Should have spent more time to rephrase and explicitly say so.
Thanks for pointing that out.

planetenxin: Sorry for my previous message, was not meant to be rude.

andre
--
Andre Klapper | Bugwrangler / Developer Advocate
https://blogs.gnome.org/aklapper/



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

planetenxin
Am 19.03.2019 um 12:21 schrieb Andre Klapper:
> planetenxin: Sorry for my previous message, was not meant to be rude.

no worries. Hope, that Gerrit is back alive soon. :-)

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Alexandros Kosiaris
Gerrit is back up. Almost all of the vandalism has been cleaned up,
some minor stuff remains, we will clean that up as well.

On Tue, Mar 19, 2019 at 1:42 PM planetenxin <[hidden email]> wrote:

>
> Am 19.03.2019 um 12:21 schrieb Andre Klapper:
> > planetenxin: Sorry for my previous message, was not meant to be rude.
>
> no worries. Hope, that Gerrit is back alive soon. :-)
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Alexandros Kosiaris
Senior Site Reliability Engineer
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Fæ
Thanks to everyone who helped sort this out.

In some ways, the vandalism neatly demonstrates how Wikimedia projects
rely on trust. When these things happen, it is a nice reminder that
our open values mean that we should take a light approach to security
whenever the potential exposure is always going to be recoverable.
Resilience rather than impenetrable, for our community at least, is a
healthy way to prioritize. The occasional predictable idiot is no
reason to change that approach.

Cheers,
Fae

On Tue, 19 Mar 2019 at 13:28, Alexandros Kosiaris
<[hidden email]> wrote:

>
> Gerrit is back up. Almost all of the vandalism has been cleaned up,
> some minor stuff remains, we will clean that up as well.
>
> On Tue, Mar 19, 2019 at 1:42 PM planetenxin <[hidden email]> wrote:
> >
> > Am 19.03.2019 um 12:21 schrieb Andre Klapper:
> > > planetenxin: Sorry for my previous message, was not meant to be rude.
> >
> > no worries. Hope, that Gerrit is back alive soon. :-)
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l

--
[hidden email] https://commons.wikimedia.org/wiki/User:Fae

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Alexandros Kosiaris
Hello Fæ,

While I understand and agree with your point, I must point out that
this 4 days have been hectic on many people from multiple teams. The
amount of work to cleanup one person's destructive half hour spree is
staggering. We need better tooling for sure to combat this, something
that while MediaWiki is already equipped with, some of the
infrastructure tools are not (yet hopefully). It saddens me greatly to
say that, but we might have to take some steps in the opposite
direction, for a while at least, until we are in shape to combat this
more effectively.

Regards,

On Tue, Mar 19, 2019 at 3:40 PM Fæ <[hidden email]> wrote:

>
> Thanks to everyone who helped sort this out.
>
> In some ways, the vandalism neatly demonstrates how Wikimedia projects
> rely on trust. When these things happen, it is a nice reminder that
> our open values mean that we should take a light approach to security
> whenever the potential exposure is always going to be recoverable.
> Resilience rather than impenetrable, for our community at least, is a
> healthy way to prioritize. The occasional predictable idiot is no
> reason to change that approach.
>
> Cheers,
> Fae
>
> On Tue, 19 Mar 2019 at 13:28, Alexandros Kosiaris
> <[hidden email]> wrote:
> >
> > Gerrit is back up. Almost all of the vandalism has been cleaned up,
> > some minor stuff remains, we will clean that up as well.
> >
> > On Tue, Mar 19, 2019 at 1:42 PM planetenxin <[hidden email]> wrote:
> > >
> > > Am 19.03.2019 um 12:21 schrieb Andre Klapper:
> > > > planetenxin: Sorry for my previous message, was not meant to be rude.
> > >
> > > no worries. Hope, that Gerrit is back alive soon. :-)
> > >
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > [hidden email]
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> --
> [hidden email] https://commons.wikimedia.org/wiki/User:Fae
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Alexandros Kosiaris
Senior Site Reliability Engineer
Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Pine W
I'd like to give a quick thanks to folks who have been dealing with
turbulence.

I think that short term mitigation measures sound reasonable, while longer
term improvements are planned and developed.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Gerrit outage

Lars Wirzenius
On Wed, Mar 20, 2019 at 06:56:19PM +0000, Pine W wrote:
> I'd like to give a quick thanks to folks who have been dealing with
> turbulence.

Me too. It's inspirational and awe-inspiring to see people work on
this.

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l