New committer: Russ Nelson

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

New committer: Russ Nelson

Rob Lanphier
Hi everyone,

Russ Nelson (svn account: nelson) is a new committer to core
MediaWiki.  Russ is contracting to Wikimedia Foundation to create a
scalable media storage system based on OpenStack's Swift object store,
some Swift middleware custom for MediaWiki, and File and
FileRepo-derived classes specific to Swift.

User page:
http://www.mediawiki.org/wiki/User:RussNelson

Rob

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: New committer: Russ Nelson

Daniel Friesen-4
Take a look at riak too. Wikia's sysadmin Artur (crucially) built file
storage for Wikia using riak and a custom fuse module.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


On 11-03-18 04:09 PM, Rob Lanphier wrote:

> Hi everyone,
>
> Russ Nelson (svn account: nelson) is a new committer to core
> MediaWiki.  Russ is contracting to Wikimedia Foundation to create a
> scalable media storage system based on OpenStack's Swift object store,
> some Swift middleware custom for MediaWiki, and File and
> FileRepo-derived classes specific to Swift.
>
> User page:
> http://www.mediawiki.org/wiki/User:RussNelson
>
> Rob

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: New committer: Russ Nelson

Antoine Musso-3
In reply to this post by Rob Lanphier
On 19/03/11 00:09, Rob Lanphier wrote:
> Hi everyone,
>
> Russ Nelson (svn account: nelson) is a new committer to core
> MediaWiki.  Russ is contracting to Wikimedia Foundation to create a
> scalable media storage system based on OpenStack's Swift object store,
> some Swift middleware custom for MediaWiki, and File and
> FileRepo-derived classes specific to Swift.

Welcome around Russ!

--
Ashar Voultoiz


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: New committer: Russ Nelson

Neil Kandalgaonkar
In reply to this post by Daniel Friesen-4
On 3/18/11 9:22 PM, Daniel Friesen wrote:
> Take a look at riak too. Wikia's sysadmin Artur (crucially) built file
> storage for Wikia using riak and a custom fuse module.

This is way past the planning stages -- Russell has been working on this
since late last year, along with a handful of WMF people.

Early on we decided we wanted a REST interface more than a FUSE
interface, so we had a different set of choices. Swift not only seems to
perform well, but it also has some nice management consoles built-in,
and it's backed by some large companies.

I don't think Riak ever came up in the discussions -- it might have been
eliminated early due to being Erlang based. While Erlang is a wicked
cool language, we have zero experience with it in production.

I had a look at Artur's RiakFuse just now. As far as I can tell it works
like this:

     MediaWiki (PHP) --> FUSE --> RiakFuse (Perl) --> HTTP REST/JSON -->
Riak (Erlang)

Artur's a lot smarter than I am... but I look at this and am already a
bit alarmed. ;)

Also I wonder why not go directly from a MediaWiki FileRepo using REST
calls straight into Riak. That's more or less what Russell is building
for us except the backend will be Swift.

Also, in the README Artur mentions his solution isn't good for big
files, and the whole point of this project was a backend for Commons.

I can read Perl fine but everybody else hated the idea of using Perl
solutions, so perhaps Artur's idea was doubly doomed from our
perspective. Still I feel bad that we didn't at least consider this
earlier, unless Russell did at some point and rejected it early on. (?)

--
Neil Kandalgaonkar (|  <[hidden email]>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: New committer: Russ Nelson

Daniel Friesen-4
On 11-03-22 01:22 AM, Neil Kandalgaonkar wrote:
> [...]
> Also I wonder why not go directly from a MediaWiki FileRepo using REST
> calls straight into Riak. That's more or less what Russell is building
> for us except the backend will be Swift.
> [...]
I actually asked Artur about that myself, I was contemplating building a
FileRepo store using GridFS (since one of my potential environments
would already have distributed MongoDB theoretically).
iirc it was something to do with different edge cases popping up and
causing bugs for anything not filesystem based. Also, iirc most of the
FileRepo related to thumbnailing actions is fairly FS geared. I did
check and found that most of the programs can handle fetching images
over http but that was probably one of the big sources of edge cases and
bugs.
That being said, since this is a Wikimedia project having the FileRepo
cleaned up so that non-fs based file storage is a viable option without
buggyness would be beautiful.

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l