wikitweets: view tweets that reference wikipedia in realtime

[Ed Summers]

This is more on the experimental side of "research" but I just
finished a prototype realtime visualization of tweets that reference
Wikipedia:

    http://wikitweets.herokuapp.com/

wikitweets is a NodeJS [1] application that listens to the Twitter
Streaming API [2] for tweets that contain Wikipedia URLs, and then
looks up the relevant Wikipedia article using the API to ultimately
stream the information to the browser using SocketIO [3]. The most
amazing thing for me is seeing the application run comfortably (so
far) as a single process on Heroku with no attached database needed.
If you are curious the code is on GitHub [4].

The key to wikistream working at all is that Twitter allows you to
search and filter the stream using the original (unshorted) URL. So
for example a Tweet with the text:

    Question of the Day: What’s the greatest seafaring movie ever?
    Some suggestions: http://bit.ly/IqsE1e (But anything on water'll work)
    #QOD [5]

Is discoverable with a search query like:

    Question of the Day wikipedia.org [6]

Note "wikipedia.org" doesn't exist in the text of the original tweet
at all, since it has been shortened by bit.ly -- but it is still
searchable because Twitter appear to be unshortening and indexing
URLs. Anyhow, I thought I'd share here since this also relied heavily
on the various language Wikipedia APIs.

//Ed

[1] http://nodejs.org
[2] https://dev.twitter.com/docs/streaming-api/methods
[3] http://socket.io
[4] https://github.com/edsu/wikitweets
[5] https://twitter.com/#!/EWeitzman/status/195520487357558784
[6] https://twitter.com/#!/search/realtime/Question%20of%20the%20Day%20wikipedia.org

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Erik Moeller-4]

On Thu, Apr 26, 2012 at 8:02 AM, Ed Summers <[hidden email]> wrote:
> This is more on the experimental side of "research" but I just
> finished a prototype realtime visualization of tweets that reference
> Wikipedia:
>
>    http://wikitweets.herokuapp.com/

That's pretty nicely done and a great snapshot of the collective
Wikipedia-consciousness - thanks for sharing!
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[emijrp]

2012/4/26 Ed Summers <[hidden email]>

This is more on the experimental side of "research" but I just
finished a prototype realtime visualization of tweets that reference
Wikipedia:

   http://wikitweets.herokuapp.com/

Very cool. Do you archive the tweets or they are discarded?

--

Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com

Pre-doctoral student at the University of Cádiz (Spain)

Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam

Personal website: https://sites.google.com/site/emijrp/

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Taha Yasseri]

My appreciation too. and the same question, do you also store the records?

bests,
.t

On Thu, Apr 26, 2012 at 7:14 PM, emijrp <[hidden email]> wrote:

2012/4/26 Ed Summers <[hidden email]>

This is more on the experimental side of "research" but I just
finished a prototype realtime visualization of tweets that reference
Wikipedia:

   http://wikitweets.herokuapp.com/

Very cool. Do you archive the tweets or they are discarded?

--

Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com

Pre-doctoral student at the University of Cádiz (Spain)

Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam

Personal website: https://sites.google.com/site/emijrp/

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

--
Taha.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Ed Summers]

At the moment there is no data store at play at all in the deployment
on Heroku. Data is simply streamed from Twitter and Wikipedia and then
delivered to any browsers who happens to be listening. It would be
trivial to add a persistence layer, and some way of making the data
available if there s a perceived need for the data. So it sounds like
you might be interested?

On Thu, Apr 26, 2012 at 1:14 PM, emijrp <[hidden email]> wrote:

> 2012/4/26 Ed Summers <[hidden email]>
>>
>> This is more on the experimental side of "research" but I just
>> finished a prototype realtime visualization of tweets that reference
>> Wikipedia:
>>
>>    http://wikitweets.herokuapp.com/
>>
>
> Very cool. Do you archive the tweets or they are discarded?
>
> --
> Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com
> Pre-doctoral student at the University of Cádiz (Spain)
> Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam
> Personal website: https://sites.google.com/site/emijrp/
>
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Taha Yasseri]

I would be very much interested, since such data shows us when and where people refer to WP in an overall image.

On Fri, Apr 27, 2012 at 3:16 AM, Ed Summers <[hidden email]> wrote:

At the moment there is no data store at play at all in the deployment
on Heroku. Data is simply streamed from Twitter and Wikipedia and then
delivered to any browsers who happens to be listening. It would be
trivial to add a persistence layer, and some way of making the data
available if there s a perceived need for the data. So it sounds like
you might be interested?

On Thu, Apr 26, 2012 at 1:14 PM, emijrp <[hidden email]> wrote:

> 2012/4/26 Ed Summers <[hidden email]>
>>
>> This is more on the experimental side of "research" but I just
>> finished a prototype realtime visualization of tweets that reference
>> Wikipedia:
>>
>>    http://wikitweets.herokuapp.com/
>>
>
> Very cool. Do you archive the tweets or they are discarded?
>
> --
> Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com
> Pre-doctoral student at the University of Cádiz (Spain)
> Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam
> Personal website: https://sites.google.com/site/emijrp/
>
>

> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

--
Taha.

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Federico Leva (Nemo)]

Taha Yasseri, 27/04/2012 04:45:
> I would be very much interested, since such data shows us when and where
> people refer to WP in an overall image.

Archiving tweets is surely useful, cf.
<http://archive.org/details/archiveteam-json-twitterstream-2012>. An
archive.org item with "wikitweets" in whatever format you grab them in
(JSON? RSS?) should be quite trivial to do, and much appreciated.

Nemo

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Ed Summers]

On Sat, Apr 28, 2012 at 5:08 AM, Federico Leva (Nemo)
<[hidden email]> wrote:
> Archiving tweets is surely useful, cf.
> <http://archive.org/details/archiveteam-json-twitterstream-2012>. An
> archive.org item with "wikitweets" in whatever format you grab them in
> (JSON? RSS?) should be quite trivial to do, and much appreciated.

Good idea. I hadn't thought of using archive.org for this, and have
been actually looking for an excuse to upload something to
archive.org. Relying on the local filesystem on heroku is somewhat
problematic [1], so I will probably need to run this archiving process
elsewhere. Perhaps I'll get it running in wikimedia labs. Thanks for
the suggestion!

//Ed

[1] https://devcenter.heroku.com/articles/java-faq#can_i_read_from_and_write_to_the_file_system

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Ed Summers]

Emilio, Taha:

I realize this was long enough ago that you may no longer be
interested but I finally got around to adding an archive function to
wikitweets [1]. Every time the app collects 1000 tweets that reference
Wikipedia it dumps them to a file on Internet Archive [2].

One nice side effect of this is that you get a BitTorrent seed/peer
for free [3], which makes mirroring the data pretty simple...if you
have a BitTorrent client handy. I blogged a little bit about how it
the archive function in wikitweets works [4].

Best,
//Ed

[1] http://wikitweets.herokuapp.com
[2] http://archive.org/download/wikitweets/wikitweets_archive.torrent
[3] http://archive.org/download/wikitweets/wikitweets_archive.torrent
[4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/

On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri <[hidden email]> wrote:

> My appreciation too. and the same question, do you also store the records?
>
> bests,
> .t
>
> On Thu, Apr 26, 2012 at 7:14 PM, emijrp <[hidden email]> wrote:
>>
>> 2012/4/26 Ed Summers <[hidden email]>
>>>
>>> This is more on the experimental side of "research" but I just
>>> finished a prototype realtime visualization of tweets that reference
>>> Wikipedia:
>>>
>>>    http://wikitweets.herokuapp.com/
>>>
>>
>> Very cool. Do you archive the tweets or they are discarded?
>>
>> --
>> Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com
>> Pre-doctoral student at the University of Cádiz (Spain)
>> Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam
>> Personal website: https://sites.google.com/site/emijrp/
>>
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
>
>
> --
> Taha.
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Dario Taraborelli-3]

Ed,

that's awesome – do you mind adding an entry on the DataHub?

http://thedatahub.org/group/wikimedia

Dario

On Sep 19, 2012, at 6:57 PM, Ed Summers <[hidden email]> wrote:

> Emilio, Taha:
>
> I realize this was long enough ago that you may no longer be
> interested but I finally got around to adding an archive function to
> wikitweets [1]. Every time the app collects 1000 tweets that reference
> Wikipedia it dumps them to a file on Internet Archive [2].
>
> One nice side effect of this is that you get a BitTorrent seed/peer
> for free [3], which makes mirroring the data pretty simple...if you
> have a BitTorrent client handy. I blogged a little bit about how it
> the archive function in wikitweets works [4].
>
> Best,
> //Ed
>
> [1] http://wikitweets.herokuapp.com
> [2] http://archive.org/download/wikitweets/wikitweets_archive.torrent
> [3] http://archive.org/download/wikitweets/wikitweets_archive.torrent
> [4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/
>
> On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri <[hidden email]> wrote:
>> My appreciation too. and the same question, do you also store the records?
>>
>> bests,
>> .t
>>
>> On Thu, Apr 26, 2012 at 7:14 PM, emijrp <[hidden email]> wrote:
>>>
>>> 2012/4/26 Ed Summers <[hidden email]>
>>>>
>>>> This is more on the experimental side of "research" but I just
>>>> finished a prototype realtime visualization of tweets that reference
>>>> Wikipedia:
>>>>
>>>>   http://wikitweets.herokuapp.com/
>>>>
>>>
>>> Very cool. Do you archive the tweets or they are discarded?
>>>
>>> --
>>> Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com
>>> Pre-doctoral student at the University of Cádiz (Spain)
>>> Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam
>>> Personal website: https://sites.google.com/site/emijrp/
>>>
>>>
>>> _______________________________________________
>>> Wiki-research-l mailing list
>>> [hidden email]
>>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>>
>>
>>
>>
>> --
>> Taha.
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Samuel Klein-2]

I love this tool so very much :)   thank you!

On Wed, Sep 19, 2012 at 9:57 PM, Ed Summers <[hidden email]> wrote:

Emilio, Taha:

I realize this was long enough ago that you may no longer be
interested but I finally got around to adding an archive function to
wikitweets [1]. Every time the app collects 1000 tweets that reference
Wikipedia it dumps them to a file on Internet Archive [2].

One nice side effect of this is that you get a BitTorrent seed/peer
for free [3], which makes mirroring the data pretty simple...if you
have a BitTorrent client handy. I blogged a little bit about how it
the archive function in wikitweets works [4].

Best,
//Ed

[1] http://wikitweets.herokuapp.com
[2] http://archive.org/download/wikitweets/wikitweets_archive.torrent
[3] http://archive.org/download/wikitweets/wikitweets_archive.torrent
[4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/

On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri <[hidden email]> wrote:
> My appreciation too. and the same question, do you also store the records?
>
> bests,
> .t
>
> On Thu, Apr 26, 2012 at 7:14 PM, emijrp <[hidden email]> wrote:
>>
>> 2012/4/26 Ed Summers <[hidden email]>
>>>
>>> This is more on the experimental side of "research" but I just
>>> finished a prototype realtime visualization of tweets that reference
>>> Wikipedia:
>>>
>>>    http://wikitweets.herokuapp.com/
>>>
>>
>> Very cool. Do you archive the tweets or they are discarded?
>>
>> --
>> Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com
>> Pre-doctoral student at the University of Cádiz (Spain)
>> Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam
>> Personal website: https://sites.google.com/site/emijrp/
>>
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
>
>
> --
> Taha.
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

--
Samuel Klein          @metasj           w:user:sj          [phone]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

[Taha Yasseri]

Thank you Ed,
That's something definitely worth-studying. Especially in order to have a less biased monitoring tool for "collective attention".

On Thu, Sep 20, 2012 at 10:23 AM, Samuel Klein <[hidden email]> wrote:

I love this tool so very much :)   thank you!

On Wed, Sep 19, 2012 at 9:57 PM, Ed Summers <[hidden email]> wrote:

Emilio, Taha:

I realize this was long enough ago that you may no longer be
interested but I finally got around to adding an archive function to
wikitweets [1]. Every time the app collects 1000 tweets that reference
Wikipedia it dumps them to a file on Internet Archive [2].

One nice side effect of this is that you get a BitTorrent seed/peer
for free [3], which makes mirroring the data pretty simple...if you
have a BitTorrent client handy. I blogged a little bit about how it
the archive function in wikitweets works [4].

Best,
//Ed

[1] http://wikitweets.herokuapp.com
[2] http://archive.org/download/wikitweets/wikitweets_archive.torrent
[3] http://archive.org/download/wikitweets/wikitweets_archive.torrent
[4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/

On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri <[hidden email]> wrote:
> My appreciation too. and the same question, do you also store the records?
>
> bests,
> .t
>
> On Thu, Apr 26, 2012 at 7:14 PM, emijrp <[hidden email]> wrote:
>>
>> 2012/4/26 Ed Summers <[hidden email]>
>>>
>>> This is more on the experimental side of "research" but I just
>>> finished a prototype realtime visualization of tweets that reference
>>> Wikipedia:
>>>
>>>    http://wikitweets.herokuapp.com/
>>>
>>
>> Very cool. Do you archive the tweets or they are discarded?
>>
>> --
>> Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com
>> Pre-doctoral student at the University of Cádiz (Spain)
>> Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam
>> Personal website: https://sites.google.com/site/emijrp/
>>
>>
>> _______________________________________________
>> Wiki-research-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>>
>
>
>
> --
> Taha.
>
> _______________________________________________
> Wiki-research-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
>

_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

--
Samuel Klein          @metasj           w:user:sj          <a href="tel:%2B1%20617%20529%204266" value="[phone]" target="_blank">[phone]


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

--
.t


_______________________________________________
Wiki-research-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l