AJAX sharing of Commons data/metadata

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

AJAX sharing of Commons data/metadata

Alex Brollo
As you know, wikisource needs robust, well-defined data, and there's a
strict, deep relationship between wikisource and Commons since Commons
hosts images of books, in .djvu or .pdf files. Commons shares both images
and contents fo information page of images, so that any wiki project can
visualize a view-only "pseudo-page" accessing to a local page named as the
file name into Commons.

Working into self-made data semantization into it.wikisouce using a lot of
creative tricks, we discovered that it's hard/almost impossible  to read by
AJAX calls the contents of pages of other projects since well-known same
origin policy, but that File: local pages are considered as coming from
"same origin" so that they can be read as any other local page, and this
AJAX call asking for the content of
i.e. File:Die_Judenfrage_in_Deutchland_1936.djvu:

html=$.ajax({url:"
http://wikisource.org/wiki/File:Die_Judenfrage_in_Deutchland_1936.djvu
",async:false}).responseText;


gives back the html text of local File: view-only page, and this means that
any data stored into information page into Commons is freely accessible by
a javascript script and can easily used locally. In particular, data stored
into information and/or (much better) Book and Creator templates can be
retrieved and parsed

Has this been described/used before? It seems a plain, simple way to share
and disseminate good, consistent metadata into any project; and this runs
from today, without any change on current wiki software.

If you like, I'm sharing a practical test use of this trick into
wikisource.org too, you can import User:Alex brollo/Library.js and a lot of
smallo, original scripts will be loaded; click on "metadata" botton from
any page connected to a File: page ( namespaces Index, Page) and you'll see
a result coming from such an AJAX call.

Alex brollo, from it.wikisource
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

bawolff
On Wed, Aug 29, 2012 at 12:39 AM, Alex Brollo <[hidden email]> wrote:

> As you know, wikisource needs robust, well-defined data, and there's a
> strict, deep relationship between wikisource and Commons since Commons
> hosts images of books, in .djvu or .pdf files. Commons shares both images
> and contents fo information page of images, so that any wiki project can
> visualize a view-only "pseudo-page" accessing to a local page named as the
> file name into Commons.
>
> Working into self-made data semantization into it.wikisouce using a lot of
> creative tricks, we discovered that it's hard/almost impossible  to read by
> AJAX calls the contents of pages of other projects since well-known same
> origin policy, but that File: local pages are considered as coming from
> "same origin" so that they can be read as any other local page, and this
> AJAX call asking for the content of
> i.e. File:Die_Judenfrage_in_Deutchland_1936.djvu:
>
> html=$.ajax({url:"
> http://wikisource.org/wiki/File:Die_Judenfrage_in_Deutchland_1936.djvu
> ",async:false}).responseText;
>
>
> gives back the html text of local File: view-only page, and this means that
> any data stored into information page into Commons is freely accessible by
> a javascript script and can easily used locally. In particular, data stored
> into information and/or (much better) Book and Creator templates can be
> retrieved and parsed
>
> Has this been described/used before? It seems a plain, simple way to share
> and disseminate good, consistent metadata into any project; and this runs
> from today, without any change on current wiki software.
>[..]

You can also do this more directly using JSON with callback - Define a
function foo, and put
http://commons.wikimedia.org/w/api.php?action=parse&page=Main_Page&format=json&callback=foo
as the src of the script tag. This works for certain "safe" api
methods. In future we may support CORS which will allow full
cross-origin js requests between Wikimedia sites

However, I think the main issue is that people want to do more things
with metadata  than just retrieving the information with js. I suppose
it all boils down to use-cases. I'll admit I'm not overly familiar
with Wikisource's metadata use case.

--
-bawolff

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Yuvi Panda
On Thu, Aug 30, 2012 at 12:25 AM, bawolff <[hidden email]> wrote:
> You can also do this more directly using JSON with callback - Define a
> function foo, and put
> http://commons.wikimedia.org/w/api.php?action=parse&page=Main_Page&format=json&callback=foo
> as the src of the script tag. This works for certain "safe" api
> methods. In future we may support CORS which will allow full
> cross-origin js requests between Wikimedia sites

Has there been any discussion of CORS support in Mediawiki / WMF sites anywhere?


--
Yuvi Panda T
http://yuvi.in/blog

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Mark Holmquist-2
> Has there been any discussion of CORS support in Mediawiki / WMF sites anywhere?

There was some talk of it in bug 32890 [0] in UploadWizard, and I tried
to throw together code for it in a patchset [1], but I didn't spend much
time on it (the effort was mostly to put the code into a workable
patchset rather than just a snippet in a bug).

bawolff also commented on the patchset and linked to bug 20814 [2],
which is further discussion of the topic in broader strokes.

[0] https://bugzilla.wikimedia.org/show_bug.cgi?id=32890
[1] https://gerrit.wikimedia.org/r/#/c/9718/
[2] https://bugzilla.wikimedia.org/show_bug.cgi?id=20814

--
Mark Holmquist
Contractor, Wikimedia Foundation
[hidden email]
http://marktraceur.info

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Chris Steipp
As mentioned in those bugs, mediawiki support a basic CORS
implementation already. It looks like we haven't authorized any domain
for wmf projects though.

On Wed, Aug 29, 2012 at 12:03 PM, Mark Holmquist
<[hidden email]> wrote:

>> Has there been any discussion of CORS support in Mediawiki / WMF sites
>> anywhere?
>
>
> There was some talk of it in bug 32890 [0] in UploadWizard, and I tried to
> throw together code for it in a patchset [1], but I didn't spend much time
> on it (the effort was mostly to put the code into a workable patchset rather
> than just a snippet in a bug).
>
> bawolff also commented on the patchset and linked to bug 20814 [2], which is
> further discussion of the topic in broader strokes.
>
> [0] https://bugzilla.wikimedia.org/show_bug.cgi?id=32890
> [1] https://gerrit.wikimedia.org/r/#/c/9718/
> [2] https://bugzilla.wikimedia.org/show_bug.cgi?id=20814
>
> --
> Mark Holmquist
> Contractor, Wikimedia Foundation
> [hidden email]
> http://marktraceur.info
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Brion Vibber
On Wed, Aug 29, 2012 at 2:55 PM, Chris Steipp <[hidden email]> wrote:

> As mentioned in those bugs, mediawiki support a basic CORS
> implementation already. It looks like we haven't authorized any domain
> for wmf projects though.
>

Looks like Roan is taking charge on it on bug 20814, yay. :)

There are also open requests about setting up CORS headers to allow read
access to upload & bits:
https://bugzilla.wikimedia.org/show_bug.cgi?id=28700
https://bugzilla.wikimedia.org/show_bug.cgi?id=25886

Access for upload is important for supporting web-based tools for SVG or
raster image editing (eg deploying SVGEditor extension).

[Note that we want *other sites* to be able to load files from
upload.wikimedia.org, not files on upload.wikimedia.org to be able to open
anything else. :)]

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Alex Brollo
Thanks for comments.

Relationship between wikisource and Commons is very strict, and there's a
large 1:1 match between structured wikisource data stored into well-formed
templates (used into nsIndex and ns0) and Book template; there's too a 1:1
relationship between nsCreator  into Commons and nsAuthor  into wikisource.

Djvu and (less used) pdf files are already shared among different
wikisource projects, but data stored into information page of files are not
shared, so that any project rewrites them and stores them with a variety of
formats and contents, introducing redundance and mining deeply coherence.

So, we are going to use Commons metadata - already stored into Book and
Creator templates - to share them widely anong all projects that need them.
When metadata are uploaded and  parsed they can be used to feed local
wikisource templates and/or to align data with automated procedures.

Thanks for API suggestion, but the question is: does it violates "same
origin" AJAX policy? I can read anything by a bot from any project, but
AJAX is great to enhance interactivity and to help user just when user
needs data, i.e. in edit mode.

Yes, the solution will be CORS, but I can't wait for future enhancements
when data can be accessed and used today, with present software.

Alex brollo
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

bawolff
On Wed, Aug 29, 2012 at 2:24 PM, Alex Brollo <[hidden email]> wrote:
> Thanks for comments.
[..]
>
> Thanks for API suggestion, but the question is: does it violates "same
> origin" AJAX policy? I can read anything by a bot from any project, but
> AJAX is great to enhance interactivity and to help user just when user
> needs data, i.e. in edit mode.
>

No it doesn't violate the same origin policy. Same origin policy only
prevents reading information from other websites, it does not stop you
from executing content from other websites (Which always seemed an odd
distinction to me...). Thus you can use the api with a callback
parameter to get around the same origin policy.

Obviously CORS is a much nicer solution.

-bawolff

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Alex Brollo
2012/8/29 bawolff <[hidden email]>

> On Wed, Aug 29, 2012 at 2:24 PM, Alex Brollo <[hidden email]>
> wrote:
> > Thanks for comments.
> [..]
> >
> > Thanks for API suggestion, but the question is: does it violates "same
> > origin" AJAX policy? I can read anything by a bot from any project, but
> > AJAX is great to enhance interactivity and to help user just when user
> > needs data, i.e. in edit mode.
> >
>
> No it doesn't violate the same origin policy. Same origin policy only
> prevents reading information from other websites, it does not stop you
> from executing content from other websites (Which always seemed an odd
> distinction to me...). Thus you can use the api with a callback
> parameter to get around the same origin policy.
>
> Obviously CORS is a much nicer solution.
>
> -bawolff
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Alex Brollo
No it doesn't violate the same origin policy. Same origin policy only
>> prevents reading information from other websites, it does not stop you
>> from executing content from other websites (Which always seemed an odd
>> distinction to me...). Thus you can use the api with a callback
>> parameter to get around the same origin policy.
>>
>
Ouch.... this is a little bit above my skill & understanding (really I
discovered AJAX not far ago). . Where can I find some examples of API
inter-project data exchage wth callback parameter?

I.e: I'd like to get the raw content and the html rendering
of File:I_promessi_sposi_(1840).djvu from wikisource, with an AJAX callback
call. Which is the needed code?

:-(

Alex
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Brion Vibber
On Wed, Aug 29, 2012 at 6:03 PM, Alex Brollo <[hidden email]> wrote:

> Ouch.... this is a little bit above my skill & understanding (really I
> discovered AJAX not far ago). . Where can I find some examples of API
> inter-project data exchage wth callback parameter?
>
> I.e: I'd like to get the raw content and the html rendering
> of File:I_promessi_sposi_(1840).djvu from wikisource, with an AJAX callback
> call. Which is the needed code?
>

Luckily, if you're using jQuery much of the low-level stuff can be taken
care of for you. Something like this should work for API calls,
automatically using the callback behind the scenes:

$.ajax({
  url: 'https://commons.wikimedia.org/w/api.php',
  data: {
    format: 'json',
    // ... various settings ...
   },
   dataType: 'jsonp' // this is the important one!
}).done(function(data) {
  // do some work here
  alert(JSON.stringify(data));
});

You can look up the various parameters for different queries in the online
help at https://commons.wikimedia.org/w/api.php and/or playing with the API
sandbox at https://commons.wikimedia.org/wiki/Special:ApiSandbox

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Alex Brollo
2012/8/30 Brion Vibber <[hidden email]>

>
>
> Luckily, if you're using jQuery much of the low-level stuff can be taken
> care of for you. Something like this should work for API calls,
> automatically using the callback behind the scenes:
>

Thanks! really I tested some interproject AJAX API call  with no luck; but
I never tested a similar syntax. Can I ask for something more help using
your email, if needed (and I guess, it will...)? Obviuosly I'll share into
this thread any good result.

Unluckily, I found that field of Information and Book templates are well,
clearly tagged with informative ids, while Creator fields are not. I posted
a proposal there, but I know that editing complex templates is a hard job.

Alex
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Alex Brollo
Thanks again Brion, it runs perfectly and - strange to say - I got no hard
difficulty, just a little bit of review of API calls and of structure of
resulting formidable objects. It's really much simpler to parse original
template contents than resulting html from their expansion.... ;-) and AJAX
API calls are SO fast!

Alex
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Alex Brollo
Just to give a final feedback to this talk, that has been very useful for
my tries: woks are going on fastly, and are presently focused on alignement
of some structures templates whose data are shared between Commons and
Wikisource: Creator vs. Author; Book
vs  MediaWiki:Proofreadpage_index_template.

Thanks again to list members. :-)

Alex
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Daniel Friesen-2
In reply to this post by Mark Holmquist-2
On Wed, 29 Aug 2012 12:03:05 -0700, Mark Holmquist
<[hidden email]> wrote:

>> Has there been any discussion of CORS support in Mediawiki / WMF sites  
>> anywhere?
>
> There was some talk of it in bug 32890 [0] in UploadWizard, and I tried  
> to throw together code for it in a patchset [1], but I didn't spend much  
> time on it (the effort was mostly to put the code into a workable  
> patchset rather than just a snippet in a bug).
>
> bawolff also commented on the patchset and linked to bug 20814 [2],  
> which is further discussion of the topic in broader strokes.
>
> [0] https://bugzilla.wikimedia.org/show_bug.cgi?id=32890
> [1] https://gerrit.wikimedia.org/r/#/c/9718/
> [2] https://bugzilla.wikimedia.org/show_bug.cgi?id=20814

CORS will definitely be something we want to implement when
OAuth/SomethingLikeIt comes around.
JSONP is a horrid concept and we shouldn't encourage it's use.
And it's impossible to do write actions or proper OAuth over JSONP. So
CORS will be necessary for any pure-JS OAuth apps.
eg: 3rd party anti-spam web apps that let you open them up in your
browser, type in a wiki's url, connect with OAuth, and then make batch
cleanups, patrolling, etc...

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: AJAX sharing of Commons data/metadata

Daniel Friesen-2
In reply to this post by Mark Holmquist-2
On Wed, 29 Aug 2012 12:03:05 -0700, Mark Holmquist
<[hidden email]> wrote:

>> Has there been any discussion of CORS support in Mediawiki / WMF sites  
>> anywhere?
>
> There was some talk of it in bug 32890 [0] in UploadWizard, and I tried  
> to throw together code for it in a patchset [1], but I didn't spend much  
> time on it (the effort was mostly to put the code into a workable  
> patchset rather than just a snippet in a bug).
>
> bawolff also commented on the patchset and linked to bug 20814 [2],  
> which is further discussion of the topic in broader strokes.
>
> [0] https://bugzilla.wikimedia.org/show_bug.cgi?id=32890
> [1] https://gerrit.wikimedia.org/r/#/c/9718/
> [2] https://bugzilla.wikimedia.org/show_bug.cgi?id=20814

CORS will definitely be something we want to implement when
OAuth/SomethingLikeIt comes around.
JSONP is a horrid concept and we shouldn't encourage it's use.
And it's impossible to do write actions or proper OAuth over JSONP. So
CORS will be necessary for any pure-JS OAuth apps.
eg: 3rd party anti-spam web apps that let you open them up in your
browser, type in a wiki's url, connect with OAuth, and then make batch
cleanups, patrolling, etc...

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l