Wikimedia REST content API is now available in beta

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Wikimedia REST content API is now available in beta

Gabriel Wicke-3
Hello all,

I am happy to announce the beta release of the Wikimedia REST Content API
at

https://rest.wikimedia.org/

Each domain has its own API documentation, which is auto-generated from
Swagger API specs. For example, here is the link for the English Wikipedia:

https://rest.wikimedia.org/en.wikipedia.org/v1/?doc

At present, this API provides convenient and low-latency access to article
HTML, page metadata and content conversions between HTML and wikitext.
After extensive testing we are confident that these endpoints are ready for
production use, but have marked them as 'unstable' until we have also
validated this with production users. You can start writing applications
that depend on it now, if you aren't afraid of possible minor changes
before transitioning to 'stable' status. For the definition of the terms
‘stable’ and ‘unstable’ see https://www.mediawiki.org/wiki/API_versioning .

While general and not specific to VisualEditor, the selection of endpoints
reflects this release's focus on speeding up VisualEditor. By storing
private Parsoid round-trip information separately, we were able to reduce
the HTML size by about 40%. This in turn reduces network transfer and
processing times, which will make loading and saving with VisualEditor
faster. We are also switching from a cache to actual storage, which will
eliminate slow VisualEditor loads caused by cache misses. Other users of
Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
translation will benefit similarly.

But, we are not done yet. In the medium term, we plan to further reduce the
HTML size by separating out all read-write metadata. This should allow us
to use Parsoid HTML with its semantic markup
<https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly for
both views and editing without increasing the HTML size over the current
output. Combined with performance work in VisualEditor, this has the
potential to make switching to visual editing instantaneous and free of any
scrolling.

We are also investigating a sub-page-level edit API for micro-contributions
and very fast VisualEditor saves. HTML saves don't necessarily have to wait
for the page to re-render from wikitext, which means that we can
potentially make them faster than wikitext saves. For this to work we'll
need to minimize network transfer and processing time on both client and
server.

More generally, this API is intended to be the beginning of a multi-purpose
content API. Its implementation (RESTBase
<http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
Swagger API specification, which helps to make it straightforward to extend
the API with new entry points. The same API spec is also used to
auto-generate the aforementioned sandbox environment, complete with handy
"try it" buttons. So, please give it a try and let us know what you think!

This API is currently unmetered; we recommend that users not perform more
than 200 requests per second and may implement limitations if necessary.

I also want to use this opportunity to thank all contributors who made this
possible:

- Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
Services team worked hard to build RESTBase, and to make it as extensible
and clean as it is now.

- Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
Halsell and Mark Bergsma helped to procure and set up the Cassandra storage
cluster backing this API.

- The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
Marc Ordinas i Llopis is solving the extremely difficult task of converting
between wikitext and HTML, and built a new API that lets us retrieve and
pass in metadata separately.

- On the MediaWiki core team, Brad Jorsch quickly created a minimal
authorization API that will let us support private wikis, and Aaron Schulz,
Alex Monk and Ori Livneh built and extended the VirtualRestService that
lets VisualEditor and MediaWiki in general easily access external services.

We welcome your feedback here: https://www.mediawiki.org/wiki/Talk:RESTBase
- and in Phabricator
<https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback:>
.

Sincerely --

Gabriel Wicke

Principal Software Engineer, Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

James Forrester-4
On 10 March 2015 at 15:23, Gabriel Wicke <[hidden email]> wrote:

> Hello all,
>
> I am happy to announce the beta release of the Wikimedia REST Content API
> at
>
> https://rest.wikimedia.org/
>
> Each domain has its own API documentation, which is auto-generated from
> Swagger API specs. For example, here is the link for the English Wikipedia:
>
> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
>

​Congratulations, Services team
​, and all those who've helped you get to this point​
. This is a huge milestone and I'm so happy we've reached it. It'll be
hugely valuable for Mobile Web, Mobile Apps, VisualEditor and dozens​ of
other projects. Thank you!


​Yours,
--
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

[hidden email] | @jdforrester
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

Erik Moeller-4
On Tue, Mar 10, 2015 at 3:29 PM, James Forrester
<[hidden email]> wrote:
> Congratulations, Services team, and all those who've helped you get to this point.
> This is a huge milestone and I'm so happy we've reached it. It'll be
> hugely valuable for Mobile Web, Mobile Apps, VisualEditor and dozens of
> other projects. Thank you!

Well said. Very excited about the possibilities -- and great to see
Swagger in action, as well. :)
--
Erik Möller
VP of Product & Strategy, Wikimedia Foundation

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

Arthur Richards-2
In reply to this post by James Forrester-4
Wow! Awesome to see this come to life :D Congratulations on this milestone.

On Tue, Mar 10, 2015 at 3:29 PM, James Forrester <[hidden email]>
wrote:

> On 10 March 2015 at 15:23, Gabriel Wicke <[hidden email]> wrote:
>
> > Hello all,
> >
> > I am happy to announce the beta release of the Wikimedia REST Content API
> > at
> >
> > https://rest.wikimedia.org/
> >
> > Each domain has its own API documentation, which is auto-generated from
> > Swagger API specs. For example, here is the link for the English
> Wikipedia:
> >
> > https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
> >
>
> ​Congratulations, Services team
> ​, and all those who've helped you get to this point​
> . This is a huge milestone and I'm so happy we've reached it. It'll be
> hugely valuable for Mobile Web, Mobile Apps, VisualEditor and dozens​ of
> other projects. Thank you!
>
>
> ​Yours,
> --
> James D. Forrester
> Product Manager, Editing
> Wikimedia Foundation, Inc.
>
> [hidden email] | @jdforrester
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




--
Arthur Richards
Team Practices Manager
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

Marc Ordinas i Llopis
In reply to this post by Gabriel Wicke-3
Congratulations! After all the hard work that has gone into this, it's
great to see it up and running. Besides the improvements it will allow in
existing projects, I can't wait to see the new things it will enable.

-- Marc

On Tue, Mar 10, 2015 at 11:23 PM, Gabriel Wicke <[hidden email]>
wrote:

> Hello all,
>
> I am happy to announce the beta release of the Wikimedia REST Content API
> at
>
> https://rest.wikimedia.org/
>
> Each domain has its own API documentation, which is auto-generated from
> Swagger API specs. For example, here is the link for the English Wikipedia:
>
> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
>
> At present, this API provides convenient and low-latency access to article
> HTML, page metadata and content conversions between HTML and wikitext.
> After extensive testing we are confident that these endpoints are ready for
> production use, but have marked them as 'unstable' until we have also
> validated this with production users. You can start writing applications
> that depend on it now, if you aren't afraid of possible minor changes
> before transitioning to 'stable' status. For the definition of the terms
> ‘stable’ and ‘unstable’ see https://www.mediawiki.org/wiki/API_versioning
> .
>
> While general and not specific to VisualEditor, the selection of endpoints
> reflects this release's focus on speeding up VisualEditor. By storing
> private Parsoid round-trip information separately, we were able to reduce
> the HTML size by about 40%. This in turn reduces network transfer and
> processing times, which will make loading and saving with VisualEditor
> faster. We are also switching from a cache to actual storage, which will
> eliminate slow VisualEditor loads caused by cache misses. Other users of
> Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
> translation will benefit similarly.
>
> But, we are not done yet. In the medium term, we plan to further reduce
> the HTML size by separating out all read-write metadata. This should allow
> us to use Parsoid HTML with its semantic markup
> <https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly for
> both views and editing without increasing the HTML size over the current
> output. Combined with performance work in VisualEditor, this has the
> potential to make switching to visual editing instantaneous and free of any
> scrolling.
>
> We are also investigating a sub-page-level edit API for
> micro-contributions and very fast VisualEditor saves. HTML saves don't
> necessarily have to wait for the page to re-render from wikitext, which
> means that we can potentially make them faster than wikitext saves. For
> this to work we'll need to minimize network transfer and processing time on
> both client and server.
>
> More generally, this API is intended to be the beginning of a
> multi-purpose content API. Its implementation (RESTBase
> <http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
> Swagger API specification, which helps to make it straightforward to extend
> the API with new entry points. The same API spec is also used to
> auto-generate the aforementioned sandbox environment, complete with handy
> "try it" buttons. So, please give it a try and let us know what you think!
>
> This API is currently unmetered; we recommend that users not perform more
> than 200 requests per second and may implement limitations if necessary.
>
> I also want to use this opportunity to thank all contributors who made
> this possible:
>
> - Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
> Services team worked hard to build RESTBase, and to make it as extensible
> and clean as it is now.
>
> - Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
> Halsell and Mark Bergsma helped to procure and set up the Cassandra storage
> cluster backing this API.
>
> - The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
> Marc Ordinas i Llopis is solving the extremely difficult task of converting
> between wikitext and HTML, and built a new API that lets us retrieve and
> pass in metadata separately.
>
> - On the MediaWiki core team, Brad Jorsch quickly created a minimal
> authorization API that will let us support private wikis, and Aaron Schulz,
> Alex Monk and Ori Livneh built and extended the VirtualRestService that
> lets VisualEditor and MediaWiki in general easily access external services.
>
> We welcome your feedback here:
> https://www.mediawiki.org/wiki/Talk:RESTBase - and in Phabricator
> <https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback:>
> .
>
> Sincerely --
>
> Gabriel Wicke
>
> Principal Software Engineer, Wikimedia Foundation
>
> _______________________________________________
> Engineering mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

Hay (Husky)
This is awesome. Congratulations to Gabriel and the rest of the team.
I'll surely hope this will provide a stable platform for getting
Wikimedia content on even more platforms.

-- Hay

On Wed, Mar 11, 2015 at 12:20 PM, Marc Ordinas i Llopis
<[hidden email]> wrote:

> Congratulations! After all the hard work that has gone into this, it's
> great to see it up and running. Besides the improvements it will allow in
> existing projects, I can't wait to see the new things it will enable.
>
> -- Marc
>
> On Tue, Mar 10, 2015 at 11:23 PM, Gabriel Wicke <[hidden email]>
> wrote:
>
>> Hello all,
>>
>> I am happy to announce the beta release of the Wikimedia REST Content API
>> at
>>
>> https://rest.wikimedia.org/
>>
>> Each domain has its own API documentation, which is auto-generated from
>> Swagger API specs. For example, here is the link for the English Wikipedia:
>>
>> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
>>
>> At present, this API provides convenient and low-latency access to article
>> HTML, page metadata and content conversions between HTML and wikitext.
>> After extensive testing we are confident that these endpoints are ready for
>> production use, but have marked them as 'unstable' until we have also
>> validated this with production users. You can start writing applications
>> that depend on it now, if you aren't afraid of possible minor changes
>> before transitioning to 'stable' status. For the definition of the terms
>> ‘stable’ and ‘unstable’ see https://www.mediawiki.org/wiki/API_versioning
>> .
>>
>> While general and not specific to VisualEditor, the selection of endpoints
>> reflects this release's focus on speeding up VisualEditor. By storing
>> private Parsoid round-trip information separately, we were able to reduce
>> the HTML size by about 40%. This in turn reduces network transfer and
>> processing times, which will make loading and saving with VisualEditor
>> faster. We are also switching from a cache to actual storage, which will
>> eliminate slow VisualEditor loads caused by cache misses. Other users of
>> Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
>> translation will benefit similarly.
>>
>> But, we are not done yet. In the medium term, we plan to further reduce
>> the HTML size by separating out all read-write metadata. This should allow
>> us to use Parsoid HTML with its semantic markup
>> <https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly for
>> both views and editing without increasing the HTML size over the current
>> output. Combined with performance work in VisualEditor, this has the
>> potential to make switching to visual editing instantaneous and free of any
>> scrolling.
>>
>> We are also investigating a sub-page-level edit API for
>> micro-contributions and very fast VisualEditor saves. HTML saves don't
>> necessarily have to wait for the page to re-render from wikitext, which
>> means that we can potentially make them faster than wikitext saves. For
>> this to work we'll need to minimize network transfer and processing time on
>> both client and server.
>>
>> More generally, this API is intended to be the beginning of a
>> multi-purpose content API. Its implementation (RESTBase
>> <http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
>> Swagger API specification, which helps to make it straightforward to extend
>> the API with new entry points. The same API spec is also used to
>> auto-generate the aforementioned sandbox environment, complete with handy
>> "try it" buttons. So, please give it a try and let us know what you think!
>>
>> This API is currently unmetered; we recommend that users not perform more
>> than 200 requests per second and may implement limitations if necessary.
>>
>> I also want to use this opportunity to thank all contributors who made
>> this possible:
>>
>> - Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
>> Services team worked hard to build RESTBase, and to make it as extensible
>> and clean as it is now.
>>
>> - Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
>> Halsell and Mark Bergsma helped to procure and set up the Cassandra storage
>> cluster backing this API.
>>
>> - The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
>> Marc Ordinas i Llopis is solving the extremely difficult task of converting
>> between wikitext and HTML, and built a new API that lets us retrieve and
>> pass in metadata separately.
>>
>> - On the MediaWiki core team, Brad Jorsch quickly created a minimal
>> authorization API that will let us support private wikis, and Aaron Schulz,
>> Alex Monk and Ori Livneh built and extended the VirtualRestService that
>> lets VisualEditor and MediaWiki in general easily access external services.
>>
>> We welcome your feedback here:
>> https://www.mediawiki.org/wiki/Talk:RESTBase - and in Phabricator
>> <https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback:>
>> .
>>
>> Sincerely --
>>
>> Gabriel Wicke
>>
>> Principal Software Engineer, Wikimedia Foundation
>>
>> _______________________________________________
>> Engineering mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>
>>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

Derk-Jan Hartman
Awesome, can't wait to see the 'downstream' effects of such an impressive
new piece in our software puzzle globe.

Great work team

DJ

On Wed, Mar 11, 2015 at 1:00 PM, Hay (Husky) <[hidden email]> wrote:

> This is awesome. Congratulations to Gabriel and the rest of the team.
> I'll surely hope this will provide a stable platform for getting
> Wikimedia content on even more platforms.
>
> -- Hay
>
> On Wed, Mar 11, 2015 at 12:20 PM, Marc Ordinas i Llopis
> <[hidden email]> wrote:
> > Congratulations! After all the hard work that has gone into this, it's
> > great to see it up and running. Besides the improvements it will allow in
> > existing projects, I can't wait to see the new things it will enable.
> >
> > -- Marc
> >
> > On Tue, Mar 10, 2015 at 11:23 PM, Gabriel Wicke <[hidden email]>
> > wrote:
> >
> >> Hello all,
> >>
> >> I am happy to announce the beta release of the Wikimedia REST Content
> API
> >> at
> >>
> >> https://rest.wikimedia.org/
> >>
> >> Each domain has its own API documentation, which is auto-generated from
> >> Swagger API specs. For example, here is the link for the English
> Wikipedia:
> >>
> >> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
> >>
> >> At present, this API provides convenient and low-latency access to
> article
> >> HTML, page metadata and content conversions between HTML and wikitext.
> >> After extensive testing we are confident that these endpoints are ready
> for
> >> production use, but have marked them as 'unstable' until we have also
> >> validated this with production users. You can start writing applications
> >> that depend on it now, if you aren't afraid of possible minor changes
> >> before transitioning to 'stable' status. For the definition of the terms
> >> ‘stable’ and ‘unstable’ see
> https://www.mediawiki.org/wiki/API_versioning
> >> .
> >>
> >> While general and not specific to VisualEditor, the selection of
> endpoints
> >> reflects this release's focus on speeding up VisualEditor. By storing
> >> private Parsoid round-trip information separately, we were able to
> reduce
> >> the HTML size by about 40%. This in turn reduces network transfer and
> >> processing times, which will make loading and saving with VisualEditor
> >> faster. We are also switching from a cache to actual storage, which will
> >> eliminate slow VisualEditor loads caused by cache misses. Other users of
> >> Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
> >> translation will benefit similarly.
> >>
> >> But, we are not done yet. In the medium term, we plan to further reduce
> >> the HTML size by separating out all read-write metadata. This should
> allow
> >> us to use Parsoid HTML with its semantic markup
> >> <https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly
> for
> >> both views and editing without increasing the HTML size over the current
> >> output. Combined with performance work in VisualEditor, this has the
> >> potential to make switching to visual editing instantaneous and free of
> any
> >> scrolling.
> >>
> >> We are also investigating a sub-page-level edit API for
> >> micro-contributions and very fast VisualEditor saves. HTML saves don't
> >> necessarily have to wait for the page to re-render from wikitext, which
> >> means that we can potentially make them faster than wikitext saves. For
> >> this to work we'll need to minimize network transfer and processing
> time on
> >> both client and server.
> >>
> >> More generally, this API is intended to be the beginning of a
> >> multi-purpose content API. Its implementation (RESTBase
> >> <http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
> >> Swagger API specification, which helps to make it straightforward to
> extend
> >> the API with new entry points. The same API spec is also used to
> >> auto-generate the aforementioned sandbox environment, complete with
> handy
> >> "try it" buttons. So, please give it a try and let us know what you
> think!
> >>
> >> This API is currently unmetered; we recommend that users not perform
> more
> >> than 200 requests per second and may implement limitations if necessary.
> >>
> >> I also want to use this opportunity to thank all contributors who made
> >> this possible:
> >>
> >> - Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
> >> Services team worked hard to build RESTBase, and to make it as
> extensible
> >> and clean as it is now.
> >>
> >> - Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
> >> Halsell and Mark Bergsma helped to procure and set up the Cassandra
> storage
> >> cluster backing this API.
> >>
> >> - The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
> >> Marc Ordinas i Llopis is solving the extremely difficult task of
> converting
> >> between wikitext and HTML, and built a new API that lets us retrieve and
> >> pass in metadata separately.
> >>
> >> - On the MediaWiki core team, Brad Jorsch quickly created a minimal
> >> authorization API that will let us support private wikis, and Aaron
> Schulz,
> >> Alex Monk and Ori Livneh built and extended the VirtualRestService that
> >> lets VisualEditor and MediaWiki in general easily access external
> services.
> >>
> >> We welcome your feedback here:
> >> https://www.mediawiki.org/wiki/Talk:RESTBase - and in Phabricator
> >> <
> https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback
> :>
> >> .
> >>
> >> Sincerely --
> >>
> >> Gabriel Wicke
> >>
> >> Principal Software Engineer, Wikimedia Foundation
> >>
> >> _______________________________________________
> >> Engineering mailing list
> >> [hidden email]
> >> https://lists.wikimedia.org/mailman/listinfo/engineering
> >>
> >>
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Engineering] Wikimedia REST content API is now available in beta

Brian Gerstle
In reply to this post by Gabriel Wicke-3
Really impressive work guys, love the generated docs!

On Tue, Mar 10, 2015 at 6:23 PM, Gabriel Wicke <[hidden email]> wrote:

> Hello all,
>
> I am happy to announce the beta release of the Wikimedia REST Content API
> at
>
> https://rest.wikimedia.org/
>
> Each domain has its own API documentation, which is auto-generated from
> Swagger API specs. For example, here is the link for the English Wikipedia:
>
> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
>
> At present, this API provides convenient and low-latency access to article
> HTML, page metadata and content conversions between HTML and wikitext.
> After extensive testing we are confident that these endpoints are ready for
> production use, but have marked them as 'unstable' until we have also
> validated this with production users. You can start writing applications
> that depend on it now, if you aren't afraid of possible minor changes
> before transitioning to 'stable' status. For the definition of the terms
> ‘stable’ and ‘unstable’ see https://www.mediawiki.org/wiki/API_versioning
> .
>
> While general and not specific to VisualEditor, the selection of endpoints
> reflects this release's focus on speeding up VisualEditor. By storing
> private Parsoid round-trip information separately, we were able to reduce
> the HTML size by about 40%. This in turn reduces network transfer and
> processing times, which will make loading and saving with VisualEditor
> faster. We are also switching from a cache to actual storage, which will
> eliminate slow VisualEditor loads caused by cache misses. Other users of
> Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
> translation will benefit similarly.
>
> But, we are not done yet. In the medium term, we plan to further reduce
> the HTML size by separating out all read-write metadata. This should allow
> us to use Parsoid HTML with its semantic markup
> <https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly for
> both views and editing without increasing the HTML size over the current
> output. Combined with performance work in VisualEditor, this has the
> potential to make switching to visual editing instantaneous and free of any
> scrolling.
>
> We are also investigating a sub-page-level edit API for
> micro-contributions and very fast VisualEditor saves. HTML saves don't
> necessarily have to wait for the page to re-render from wikitext, which
> means that we can potentially make them faster than wikitext saves. For
> this to work we'll need to minimize network transfer and processing time on
> both client and server.
>
> More generally, this API is intended to be the beginning of a
> multi-purpose content API. Its implementation (RESTBase
> <http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
> Swagger API specification, which helps to make it straightforward to extend
> the API with new entry points. The same API spec is also used to
> auto-generate the aforementioned sandbox environment, complete with handy
> "try it" buttons. So, please give it a try and let us know what you think!
>
> This API is currently unmetered; we recommend that users not perform more
> than 200 requests per second and may implement limitations if necessary.
>
> I also want to use this opportunity to thank all contributors who made
> this possible:
>
> - Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
> Services team worked hard to build RESTBase, and to make it as extensible
> and clean as it is now.
>
> - Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
> Halsell and Mark Bergsma helped to procure and set up the Cassandra storage
> cluster backing this API.
>
> - The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
> Marc Ordinas i Llopis is solving the extremely difficult task of converting
> between wikitext and HTML, and built a new API that lets us retrieve and
> pass in metadata separately.
>
> - On the MediaWiki core team, Brad Jorsch quickly created a minimal
> authorization API that will let us support private wikis, and Aaron Schulz,
> Alex Monk and Ori Livneh built and extended the VirtualRestService that
> lets VisualEditor and MediaWiki in general easily access external services.
>
> We welcome your feedback here:
> https://www.mediawiki.org/wiki/Talk:RESTBase - and in Phabricator
> <https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback:>
> .
>
> Sincerely --
>
> Gabriel Wicke
>
> Principal Software Engineer, Wikimedia Foundation
>
> _______________________________________________
> Engineering mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
>


--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimedia REST content API is now available in beta

Gerard Meijssen-3
In reply to this post by Gabriel Wicke-3
Hoi,
In what way will we know how useful this is? Will we have usage statistics ?
Thanks,
      GerardM

On 10 March 2015 at 23:23, Gabriel Wicke <[hidden email]> wrote:

> Hello all,
>
> I am happy to announce the beta release of the Wikimedia REST Content API
> at
>
> https://rest.wikimedia.org/
>
> Each domain has its own API documentation, which is auto-generated from
> Swagger API specs. For example, here is the link for the English Wikipedia:
>
> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
>
> At present, this API provides convenient and low-latency access to article
> HTML, page metadata and content conversions between HTML and wikitext.
> After extensive testing we are confident that these endpoints are ready for
> production use, but have marked them as 'unstable' until we have also
> validated this with production users. You can start writing applications
> that depend on it now, if you aren't afraid of possible minor changes
> before transitioning to 'stable' status. For the definition of the terms
> ‘stable’ and ‘unstable’ see https://www.mediawiki.org/wiki/API_versioning
> .
>
> While general and not specific to VisualEditor, the selection of endpoints
> reflects this release's focus on speeding up VisualEditor. By storing
> private Parsoid round-trip information separately, we were able to reduce
> the HTML size by about 40%. This in turn reduces network transfer and
> processing times, which will make loading and saving with VisualEditor
> faster. We are also switching from a cache to actual storage, which will
> eliminate slow VisualEditor loads caused by cache misses. Other users of
> Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
> translation will benefit similarly.
>
> But, we are not done yet. In the medium term, we plan to further reduce the
> HTML size by separating out all read-write metadata. This should allow us
> to use Parsoid HTML with its semantic markup
> <https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly for
> both views and editing without increasing the HTML size over the current
> output. Combined with performance work in VisualEditor, this has the
> potential to make switching to visual editing instantaneous and free of any
> scrolling.
>
> We are also investigating a sub-page-level edit API for micro-contributions
> and very fast VisualEditor saves. HTML saves don't necessarily have to wait
> for the page to re-render from wikitext, which means that we can
> potentially make them faster than wikitext saves. For this to work we'll
> need to minimize network transfer and processing time on both client and
> server.
>
> More generally, this API is intended to be the beginning of a multi-purpose
> content API. Its implementation (RESTBase
> <http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
> Swagger API specification, which helps to make it straightforward to extend
> the API with new entry points. The same API spec is also used to
> auto-generate the aforementioned sandbox environment, complete with handy
> "try it" buttons. So, please give it a try and let us know what you think!
>
> This API is currently unmetered; we recommend that users not perform more
> than 200 requests per second and may implement limitations if necessary.
>
> I also want to use this opportunity to thank all contributors who made this
> possible:
>
> - Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
> Services team worked hard to build RESTBase, and to make it as extensible
> and clean as it is now.
>
> - Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
> Halsell and Mark Bergsma helped to procure and set up the Cassandra storage
> cluster backing this API.
>
> - The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
> Marc Ordinas i Llopis is solving the extremely difficult task of converting
> between wikitext and HTML, and built a new API that lets us retrieve and
> pass in metadata separately.
>
> - On the MediaWiki core team, Brad Jorsch quickly created a minimal
> authorization API that will let us support private wikis, and Aaron Schulz,
> Alex Monk and Ori Livneh built and extended the VirtualRestService that
> lets VisualEditor and MediaWiki in general easily access external services.
>
> We welcome your feedback here:
> https://www.mediawiki.org/wiki/Talk:RESTBase
> - and in Phabricator
> <
> https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback
> :>
> .
>
> Sincerely --
>
> Gabriel Wicke
>
> Principal Software Engineer, Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimedia REST content API is now available in beta

Gabriel Wicke-3
Hi Gerard,

On Thu, Mar 12, 2015 at 4:32 AM, Gerard Meijssen <[hidden email]>
wrote:

> Hoi,
> In what way will we know how useful this is? Will we have usage statistics
> ?


yes, we have metrics on request rates, status codes and response times by
end point. Here is a dashboard showing some of those metrics:

http://grafana.wikimedia.org/#/dashboard/db/restbase

The first users will be VisualEditor and other Parsoid clients. The
VisualEditor integration has been working out of the box on
https://test.wikipedia.org/, so the next step will be to switch
VisualEditor to use RESTBase on other phase0 wikis (notably mediawiki.org)
as well.

With all current revisions available from storage we are now also in a
position to offer HTML dumps again, which is tracked in
https://phabricator.wikimedia.org/T17017.

Gabriel
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimedia REST content API is now available in beta

rupert THURNER-2
In reply to this post by Gabriel Wicke-3
Hi Gabriel,  I am so glad to read about this excellent achievement! Is the
wikitext the original wikitext wich could be changed and saved back? And
the difference is a real difference which would allow kind of patrol
applications?

Rupert
On Mar 10, 2015 11:23 PM, "Gabriel Wicke" <[hidden email]> wrote:

> Hello all,
>
> I am happy to announce the beta release of the Wikimedia REST Content API
> at
>
> https://rest.wikimedia.org/
>
> Each domain has its own API documentation, which is auto-generated from
> Swagger API specs. For example, here is the link for the English Wikipedia:
>
> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
>
> At present, this API provides convenient and low-latency access to article
> HTML, page metadata and content conversions between HTML and wikitext.
> After extensive testing we are confident that these endpoints are ready for
> production use, but have marked them as 'unstable' until we have also
> validated this with production users. You can start writing applications
> that depend on it now, if you aren't afraid of possible minor changes
> before transitioning to 'stable' status. For the definition of the terms
> ‘stable’ and ‘unstable’ see https://www.mediawiki.org/wiki/API_versioning
> .
>
> While general and not specific to VisualEditor, the selection of endpoints
> reflects this release's focus on speeding up VisualEditor. By storing
> private Parsoid round-trip information separately, we were able to reduce
> the HTML size by about 40%. This in turn reduces network transfer and
> processing times, which will make loading and saving with VisualEditor
> faster. We are also switching from a cache to actual storage, which will
> eliminate slow VisualEditor loads caused by cache misses. Other users of
> Parsoid HTML like Flow, HTML dumps, the OCG PDF renderer or Content
> translation will benefit similarly.
>
> But, we are not done yet. In the medium term, we plan to further reduce the
> HTML size by separating out all read-write metadata. This should allow us
> to use Parsoid HTML with its semantic markup
> <https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec> directly for
> both views and editing without increasing the HTML size over the current
> output. Combined with performance work in VisualEditor, this has the
> potential to make switching to visual editing instantaneous and free of any
> scrolling.
>
> We are also investigating a sub-page-level edit API for micro-contributions
> and very fast VisualEditor saves. HTML saves don't necessarily have to wait
> for the page to re-render from wikitext, which means that we can
> potentially make them faster than wikitext saves. For this to work we'll
> need to minimize network transfer and processing time on both client and
> server.
>
> More generally, this API is intended to be the beginning of a multi-purpose
> content API. Its implementation (RESTBase
> <http://www.mediawiki.org/wiki/RESTBase>) is driven by a declarative
> Swagger API specification, which helps to make it straightforward to extend
> the API with new entry points. The same API spec is also used to
> auto-generate the aforementioned sandbox environment, complete with handy
> "try it" buttons. So, please give it a try and let us know what you think!
>
> This API is currently unmetered; we recommend that users not perform more
> than 200 requests per second and may implement limitations if necessary.
>
> I also want to use this opportunity to thank all contributors who made this
> possible:
>
> - Marko Obrovac, Eric Evans, James Douglas and Hardik Juneja on the
> Services team worked hard to build RESTBase, and to make it as extensible
> and clean as it is now.
>
> - Filippo Giunchedi, Alex Kosiaris, Andrew Otto, Faidon Liambotis, Rob
> Halsell and Mark Bergsma helped to procure and set up the Cassandra storage
> cluster backing this API.
>
> - The Parsoid team with Subbu Sastry, Arlo Breault, C. Scott Ananian and
> Marc Ordinas i Llopis is solving the extremely difficult task of converting
> between wikitext and HTML, and built a new API that lets us retrieve and
> pass in metadata separately.
>
> - On the MediaWiki core team, Brad Jorsch quickly created a minimal
> authorization API that will let us support private wikis, and Aaron Schulz,
> Alex Monk and Ori Livneh built and extended the VirtualRestService that
> lets VisualEditor and MediaWiki in general easily access external services.
>
> We welcome your feedback here:
> https://www.mediawiki.org/wiki/Talk:RESTBase
> - and in Phabricator
> <
> https://phabricator.wikimedia.org/maniphest/task/create/?projects=RESTBase&title=Feedback
> :>
> .
>
> Sincerely --
>
> Gabriel Wicke
>
> Principal Software Engineer, Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Wikimedia REST content API is now available in beta

Daniel Friesen-2
In reply to this post by Gabriel Wicke-3
On 2015-03-10 3:23 PM, Gabriel Wicke wrote:

> Hello all,
>
> I am happy to announce the beta release of the Wikimedia REST Content API
> at
>
> https://rest.wikimedia.org/
>
> Each domain has its own API documentation, which is auto-generated from
> Swagger API specs. For example, here is the link for the English Wikipedia:
>
> https://rest.wikimedia.org/en.wikipedia.org/v1/?doc
Could the RSD document EditURI points to be updated to include a
reference to an individual wiki's base path in the REST api.

ie: en.wp's RSD would have a new <api> with an
apiLink="https://rest.wikimedia.org/en.wikipedia.org/"

Ideally there would be a <docs> pointing to documentation but there's no
link that would work since <docs> shouldn't point to v1/?doc and there's
no non versioned ?doc page.


Alternately you could include -v1 in the api name="", use
apiLink="https://rest.wikimedia.org/en.wikipedia.org/v1/" and point
<docs> to `https://rest.wikimedia.org/en.wikipedia.org/v1/?doc`.

That might actually be a better idea since you won't need to provide an
API to let code performing discovery know which versions of the API are
supported.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l