WikiApiary update

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

WikiApiary update

thingles
In January I sent a note introducing WikiApiary to this list. <http://wikimedia.7.n6.nabble.com/Introducing-WikiApiary-td4993870.html> So much has happened in the last couple of months that I thought an update was worthwhile. I'm sharing this with the SMW user list since I assume being MediaWiki fans that are _also_ into SMW you would have interest. Additionally, I think WikiApiary is an interesting use of SMW itself (and is now the 9th largest SMW in it's own index with 1,187,186 properties, #1 is Malaripedia with 30,769,053 properties).

Notably:

* Over 5,000 wikis have been bulk loaded into WikiApiary.
* Website auditing and data retrieval has been (nearly) completely automated.
* Significant buildout has happened in the forms and templates making it easier to use (thanks to [[kgh]] for big help here)
* There is now a bookmarklet that you can use to easily add wikis to WikiApiary. <http://wikiapiary.com/wiki/WikiApiary:Bookmarklet>

WikiApiary is currently pulling information from 5,215 MediaWiki installs and storing hundreds of data points on each wiki in Semantic MediaWiki for easy query and display, as well as pulling statistics from each of those 5,215 wikis to create time-series data on edits, pages, users, jobs, etc. It is easiest to just see it so head over to:

http://wikiapiary.com/wiki/Main_Page

Highlights:

WikiApiary graphs itself of course (note the large import around Jan. 27 and some automation being added in beginning of March):

http://wikiapiary.com/wiki/WikiApiary

WikiApiary is the first site to ever show usage of Extensions across wikis. WikiApiary has indexed 2,352 extensions:

http://wikiapiary.com/wiki/Extension:Main_Page

...and even has a really nice drilldown for each extension:

http://wikiapiary.com/wiki/Extension:ParserFunctions

And WikiApiary allows Farms to be created and have sites added to them, as well as other farms (tree structure)

http://wikiapiary.com/wiki/Farm:Main_Page

The WikiMedia farm and it's 45 million users:

http://wikiapiary.com/wiki/Farm:Wikimedia#tab=Statistics

My wiki farm:

http://wikiapiary.com/wiki/Farm:Thingelstad.com#tab=Statistics

And this is a fun one, the largest Semantic MediaWiki sites known:

http://wikiapiary.com/wiki/Semantic_statistics

The site has three main components: the wiki with all it's templates and magic, a MySQL database specifically for the time-series data and some robots that do data collection. You can see the code for that (mostly Python) on Github:

https://github.com/thingles/WikiApiary

You can see the bots working away on the bot log:

http://wikiapiary.com/wiki/Bot_log

There is discussion of adding links from MediaWiki.org extension pages to the extension page on WikiApiary, which I think would be a great resource for people.

I've got a number of additional things to add as well as some more optimization of the bots to keep load light on the wiki. But the progress is coming along nicely and I wanted to share.

If you want to add your wiki, the simple steps are on the front page of the site.
Jamie Thingelstad
[hidden email]
mobile: 612-810-3699
find me on AIM Twitter Facebook LinkedIn

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: WikiApiary update

Jeroen De Dauw-2
Hey,

> and is now the 9th largest SMW in it's own index with 1,187,186 properties

How do you end up with so many property values? Is there a breakdown of
what you are storing in SMW somewhere?

> a MySQL database specifically for the time-series data

So I am guessing this data is no in SMW at all - correct? How are you
getting it into the timeseries visualizations? Are you using SRF somehow or
is that custom code?

Guess it'd be very interesting to SMW users if you would do a writeup of
the challenges you faced in creating this wiki and how you solved them.

> There is discussion of adding links from MediaWiki.org extension pages to
the extension page on WikiApiary, which I think would be a great resource
for people.

Awesome. Where do I go to +1 this idea?

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil.
--
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: WikiApiary update

thingles
On Mar 14, 2013, at 10:02 AM, Jeroen De Dauw <[hidden email]> wrote:

> > There is discussion of adding links from MediaWiki.org extension pages to the extension page on WikiApiary, which I think would be a great resource for people.
>
> Awesome. Where do I go to +1 this idea?

This is now live! :-)

http://wikiapiary.com/wiki/WikiApiary_talk:Operations/2013/March#Linked_to_from_Mediawiki.org.21

Jamie Thingelstad
[hidden email]
mobile: 612-810-3699
find me on AIM Twitter Facebook LinkedIn

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user