AWS usage

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

AWS usage

John Mark Vandenberg
Is AWS (Amazon Web Services) being used by Wikimedia directly; US
Foundation, or by other affiliates?

While trawling around for AWS related tasks, I saw on T74501 that Sage
Ross' team was using AWS late last year, perhaps only temporarily due
to the bug.
https://phabricator.wikimedia.org/T74501
(I assume from the date the task was raised that it was a team at Wiki
Education Foundation).

All use of Travis-CI is indirectly using AWS.  Are there other indirect uses?

Is it still in use at all in a direct manner, i.e. using our own account?

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Greg Grossmeier-2
<quote name="John Mark Vandenberg" date="2015-10-07" time="16:41:01 +1100">
> Is AWS (Amazon Web Services) being used by Wikimedia directly; US
> Foundation, or by other affiliates?

Not for anything user facing. I *think* the Performance team uses AWS
for a testing hosting (it runs Windows, I think, or I'm confusing it
with another team).

> All use of Travis-CI is indirectly using AWS.  Are there other indirect uses?

Travis-CI is only currently used by a very small number of repositories,
luckily.


Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Petr Bena
Hi,

Why do you even care? Is this question directed to foundation only or
community developers as well? "Used by wikimedia" is very broad term.



On Wed, Oct 7, 2015 at 4:42 PM, Greg Grossmeier <[hidden email]> wrote:

> <quote name="John Mark Vandenberg" date="2015-10-07" time="16:41:01 +1100">
>> Is AWS (Amazon Web Services) being used by Wikimedia directly; US
>> Foundation, or by other affiliates?
>
> Not for anything user facing. I *think* the Performance team uses AWS
> for a testing hosting (it runs Windows, I think, or I'm confusing it
> with another team).
>
>> All use of Travis-CI is indirectly using AWS.  Are there other indirect uses?
>
> Travis-CI is only currently used by a very small number of repositories,
> luckily.
>
>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

James HK
In reply to this post by Greg Grossmeier-2
Hi,

> Travis-CI is only currently used by a very small number of repositories,
> luckily.

I'm not sure I understand the underlying connotation in your comment
but some non-core developers are happy about Travis-CI [0] and how it
allows them to run a test environment independent from WMF (and/or WMF
Jenkins).

[0] https://cloud.githubusercontent.com/assets/1245473/10268587/d1b30c78-6abd-11e5-98ea-ac0531b435ce.png

Cheers

On 10/7/15, Greg Grossmeier <[hidden email]> wrote:

> <quote name="John Mark Vandenberg" date="2015-10-07" time="16:41:01 +1100">
>> Is AWS (Amazon Web Services) being used by Wikimedia directly; US
>> Foundation, or by other affiliates?
>
> Not for anything user facing. I *think* the Performance team uses AWS
> for a testing hosting (it runs Windows, I think, or I'm confusing it
> with another team).
>
>> All use of Travis-CI is indirectly using AWS.  Are there other indirect
>> uses?
>
> Travis-CI is only currently used by a very small number of repositories,
> luckily.
>
>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Greg Grossmeier-2
In reply to this post by Petr Bena
<quote name="Petr Bena" date="2015-10-07" time="16:54:47 +0200">
> Why do you even care? Is this question directed to foundation only or
> community developers as well? "Used by wikimedia" is very broad term.

I interpreted "Wikimedia" as WMF, for better or worse, in my email, just
so it's clear.

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Greg Grossmeier-2
In reply to this post by James HK
<quote name="James HK" date="2015-10-07" time="17:03:18 +0200">
> Hi,
>
> > Travis-CI is only currently used by a very small number of repositories,
> > luckily.
>
> I'm not sure I understand the underlying connotation in your comment
> but some non-core developers are happy about Travis-CI [0] and how it
> allows them to run a test environment independent from WMF (and/or WMF
> Jenkins).

As in, they would not be experiencing any outage because they are not
using Travis. The very topic of this thread.

Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Sage Ross
In reply to this post by John Mark Vandenberg
On Tue, Oct 6, 2015 at 10:41 PM, John Mark Vandenberg <[hidden email]>
wrote:

> Is AWS (Amazon Web Services) being used by Wikimedia directly; US
> Foundation, or by other affiliates?
>
> While trawling around for AWS related tasks, I saw on T74501 that Sage
> Ross' team was using AWS late last year, perhaps only temporarily due
> to the bug.
> https://phabricator.wikimedia.org/T74501
> (I assume from the date the task was raised that it was a team at Wiki
> Education Foundation).
>
>
Your assumptions here are correct: Wiki Education Foundation initially
tried using Heroku (AWS) to host an OAuth app, and then we switched to a
more traditional hosting provider that does not run on AWS.

-Sage
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Tyler Romeo
In reply to this post by Greg Grossmeier-2
On Wed, Oct 7, 2015 at 11:27 AM, Greg Grossmeier <[hidden email]> wrote:

> As in, they would not be experiencing any outage because they are not
> using Travis. The very topic of this thread.
>

Nowhere in this thread is the uptime of Travis or AWS mentioned as far as I
see, so it's understandable to be confused as to why we're "lucky" to not
be using Travis.

*-- *
*Tyler Romeo*
Stevens Institute of Technology, Class of 2016
Major in Computer Science
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Petr Bena
In reply to this post by Greg Grossmeier-2
Sorry, I mentioned the question for OP of this thread.

On Wed, Oct 7, 2015 at 5:06 PM, Greg Grossmeier <[hidden email]> wrote:

> <quote name="Petr Bena" date="2015-10-07" time="16:54:47 +0200">
>> Why do you even care? Is this question directed to foundation only or
>> community developers as well? "Used by wikimedia" is very broad term.
>
> I interpreted "Wikimedia" as WMF, for better or worse, in my email, just
> so it's clear.
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Greg Grossmeier-2
In reply to this post by Tyler Romeo
<quote name="Tyler Romeo" date="2015-10-07" time="11:41:41 -0400">
> On Wed, Oct 7, 2015 at 11:27 AM, Greg Grossmeier <[hidden email]> wrote:
>
> > As in, they would not be experiencing any outage because they are not
> > using Travis. The very topic of this thread.
> >
>
> Nowhere in this thread is the uptime of Travis or AWS mentioned as far as I
> see, so it's understandable to be confused as to why we're "lucky" to not
> be using Travis.

Hah! I thought the thread subject was "AWS outage" not "usage".

Sorry, my bad :)

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

John Mark Vandenberg
In reply to this post by Petr Bena
On Thu, Oct 8, 2015 at 1:54 AM, Petr Bena <[hidden email]> wrote:
> Hi,
>
> Why do you even care? Is this question directed to foundation only or
> community developers as well? "Used by wikimedia" is very broad term.

Pywikibot uses Travis-CI, and I was looking at their S3 artefacts
integration for storage of coverage data.
It turns out to be quite a bit more expensive than I had assumed.
https://phabricator.wikimedia.org/T74863#1710830

Also of interest was that given some people are dead against allowing
Travis-CI to do builds for free, it appears parts of our community are
using AWS directly instead of Labs, which is not free.

I used the broad term "Wikimedia", as it is the broader community that
could be using Labs.  It seems to be worth understanding why they are
choosing to not use Labs, and how much it is costing "us".

--
John Vandenberg

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Petr Bena
As to why:

AWS is more flexible and more reliable than wikimedia-labs, other than
that it's basically the same. If they really need to use AWS it's
probably because they don't like the restrictions that come with
wikimedia labs (stuff must be open source, comply with policies, only
ubuntu or debian, no proprietary software, hard-to-get public IP, long
waiting time for stuff that you can't do yourself (request new
project), no IPv6 and many others)

On Thu, Oct 8, 2015 at 6:08 AM, John Mark Vandenberg <[hidden email]> wrote:

> On Thu, Oct 8, 2015 at 1:54 AM, Petr Bena <[hidden email]> wrote:
>> Hi,
>>
>> Why do you even care? Is this question directed to foundation only or
>> community developers as well? "Used by wikimedia" is very broad term.
>
> Pywikibot uses Travis-CI, and I was looking at their S3 artefacts
> integration for storage of coverage data.
> It turns out to be quite a bit more expensive than I had assumed.
> https://phabricator.wikimedia.org/T74863#1710830
>
> Also of interest was that given some people are dead against allowing
> Travis-CI to do builds for free, it appears parts of our community are
> using AWS directly instead of Labs, which is not free.
>
> I used the broad term "Wikimedia", as it is the broader community that
> could be using Labs.  It seems to be worth understanding why they are
> choosing to not use Labs, and how much it is costing "us".
>
> --
> John Vandenberg
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AWS usage

Ryan Lane-2
On Thu, Oct 8, 2015 at 9:13 AM, Petr Bena <[hidden email]> wrote:

> As to why:
>
> AWS is more flexible and more reliable than wikimedia-labs, other than
> that it's basically the same. If they really need to use AWS it's
> probably because they don't like the restrictions that come with
> wikimedia labs (stuff must be open source, comply with policies, only
> ubuntu or debian, no proprietary software, hard-to-get public IP, long
> waiting time for stuff that you can't do yourself (request new
> project), no IPv6 and many others)
>
>
Assuming you don't use NFS, Labs is likely as reliable or possibly more
reliable than EC2, since effort is put into saving instances when hardware
is having issues (which AWS does not do). Project creation is usually
pretty fast in Labs. When you create a new account in AWS, you have to get
approved for most of the features before you can use them, so it's not
quite as quick as you're making out. Getting public IPs in Labs is
restricted, but it's also the most restricted thing in AWS too. For the
most part it's unnecessary to get/use public IPs. In AWS basically
everything goes in through ELBs and in Labs there's an ELB equivalent.

As for the rest of the restrictions, yep. Those totally make sense and fit
with the intent of Labs.

I think some of the benefits of Labs are being ignored here. You may be
limited to ubuntu/debian, but you also create an instance that's
pre-configured, that you (and any other project member) can immediately SSH
into. DNS is handled for you, load balancers are easy to use, cross-project
access is way easier (no need to manage IAM and VPCs), and networking is
pre-configured (which is actually non-trivial in AWS). Most importantly
it's free for end-users.

- Ryan
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l