SMW and Wikidata (Was: [SMW-devel] Semantic MediaWiki and Wikidata ContentHandler)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

SMW and Wikidata (Was: [SMW-devel] Semantic MediaWiki and Wikidata ContentHandler)

Markus Krötzsch-2
Hi James, hi all,

the relationship and synergies of SMW and Wikidata are of course quite
important to us, so I am taking the time to write a slightly longer
email that I also send to the user list.

At the moment, Wikidata is still under development, so it is a bit early
to make concrete plans on how to align the project.
Here are some things that I can already answer now:


(1) What is the software in "Wikidata"?

The software underlying Wikidata is called Wikibase. It consists of two
extensions: Repository (where data is stored; this will run at
wikidata.org) and Client (that pulls data into other MediaWiki sites;
this will run at wikipedia.org). Both extensions will be installable on
MediaWiki sites to get Wikidata-like features there.


(2) What are the main differences between SMW and Wikidata?

Wikidata is being built with SMW in mind, and the experiences with SMW
are a valuable input for its design. The main use case of Wikidata (a
centralised, multi-lingual site that serves as a data repository) is
different from that of SMW (a data-enhanced MediaWiki), and this leads
to a number of differences.

The data model of Wikidata is a bit more complex than that of SMW. For
example, every fact will also have an (optional) reference. Wikidata
will also use more complex datatypes (e.g., since many text values can
be translated). However, both Wikidata and SMW are still mostly based on
property-value assignments, and Wikidata will have a Property namespace.
Also, the more complex structures in Wikidata could be captured in SMW
using internal objects.

The user interface of Wikibase Repository will be based on input forms,
and thus quite different from SMW. The UI will be a bit like a
full-page, editable Factbox, but much more advanced, with a lot of
support for entering values.

It is not defined yet what kind of query language Wikidata will support
in Phase 3 (or thereafter). This needs to be designed first, but the
more complicated data model will also have an impact there.


(3) What is a ContentHandler and how could it help SMW?

ContentHandlers allow MediaWiki to manage textual content that is not
interpreted as wikitext. Wikidata uses this to store data in a
text-based format (JSON) without having to invent a MediaWiki syntax
extension. In the future, it might be possible to have "multi-part"
content for one page, so that one can store both Wikitext and other
structured data in a cleaner way.

SMW could use this to allow data to be stored directly for a page,
without having to specify the data in the wikitext. This could be a bit
like an "editable Factbox". MediaWiki could do something similar with
Categories.


(4) Will ContentHandlers reduce the dependence on custom SQL?

No. Content that is stored in textual form on wiki pages is not suitable
for answering queries quickly. To do this, it will always be necessary
to have custom tables and/or external storage engines that provide fast
access to the data. This is also the case for Wikidata.

Currently, the MW page is the *primary* storage for SMW data, while the
SQL tables are just *secondary* storage for faster lookup. The
ContentHandlers can improve primary storage handling, but won't affect
secondary storages and querying.


(5) What are the possible synergies and joint uses of Wikidata and SMW?

The details can only be worked out when Wikidata is a bit more stable
than it is now, but there are a few obvious scenarios:

* Wikibase Repository + SMW: data given in SMW annotations and data
edited in Wikidata end up in the same secondary data structures, and can
be retrieved through the same interfaces/queries. There are three main
options: SMW tables are used for Wikibase content (this would yield
query support via #ask), Wikibase tables are used for SMW content
(yields full integration with Wikibase data access features), or both.

* SMW as a Wikibase Repository: SMW could support the use of Wikibase
APIs and exports on top of its native data store, so that other sites
could fetch SMW data via the Wikidata Client.

* SMW as a Wikibase Client: SMW could also pull data from Wikibase
Repositories to use it locally and to display it in its interfaces.

* Exchanging code: Besides application-level integration, SMW and
Wikidata are also going to exchange lines of code. Initially, Wikidata
will exploit code from SMW (e.g., the calendar conversion code for times
and dates). Later, smaller pieces of Wikidata code might be used for
SMW. In many cases, this will mean that the code is adapted heavily; if
it can be used without much change, one would possibly create a "library
extension" (like Validator).


(6) Will Wikidata replace SMW in the long run?

No. If anything, the integration between both could become so good that
the distinction is no longer relevant and we are only talking about
different data authoring and data access interface extensions to
MediaWiki, which users can enable and disable at their leisure. This
will still take some time though.


(7) What is the development plan for SMW with respect to Wikidata?

The strategy is to wait until Wikidata has settled a bit before
approaching any integration. This will probably be another 10 or 11
months from now (note that data management only comes in at Phase 2 of
Wikidata, i.e., in 5 months). Exchanging some code might be possible
before that, but integrating Wikibase while it is a moving target would
not be feasible (there would not be a chance of keeping up with the
speed of Wikidata, esp. since half the SMW team is engaged in Wikidata
development right now).


I hope this answers most questions on that topic for now. Maybe it would
be nice to have a wikipage on semantic-mediawiki.org with these answers.

Best regards,

Markus


On 30/04/12 23:44, James HK wrote:

> Hi,
>
> ==WD ContentHandler ==
> Now that the Wikidata project is introducing a new ContentHandler [1],
> we where curious if in near future SMW would make use of those methods
> to store its own objects in the same way and reduces its reliance on
> non-mediawiki tables/methods.
>
> As Wikidata and SMW share the same bottlenecks in terms of
> performance, caching, scalability it might prove beneficial for SMW to
> use similar methods to store and fetch dataobjects from a mw-database
> while at the same time reduces its reliance on SQL specific code and
> lift the ban on other supported database systems(Oracle,Sqlite etc.).
>
> The reason why we are asking is that we hope that some of the work
> that has be done in SMW is reflected in Wikidata as visa versa to
> ensure that both developments share some similarities to the extend
> that their "plug and playable"  (meaning causing the least
> interruptions when installed together and make it easier for code
> reviewers to understand the inner workings which increases stability).
>
> ==SMW and WD ==
> Furthermore, with the Wikidata project on the horizon, we would be
> interested to see how SMW positioned and distinguished itself from WD
> in light of recent developments. (A simple table could help to explain
> those differences fairly quick while at the same time allows some
> directional guidance for the future of SMW.)
>
> ==Validator==
> The Validator provides validation methods for datatypes (and parameter
> definitions etc.) and WD certainly needs similar methods, is it
> planned that those methods (or part of it) are being moved into the
> core?
>
> [1] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki%2Fcore.git;a=commit;h=337f65c03b59eaddc038bc1a0fb150dc97e03154
>
> Cheers,
>
> mwjames
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Semediawiki-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: [SMW-devel] SMW and Wikidata (Was: Semantic MediaWiki and Wikidata ContentHandler)

Denny Vrandecic-3
Markus,

do you mind if we take this text as the starting point for a text detailing the relation between SMW and Wikidata here?

<https://meta.wikimedia.org/wiki/Wikidata/Notes/SMW_and_Wikidata>

Cheers,
Denny


On 1 May 2012, at 10:04, Markus Krötzsch wrote:

> Hi James, hi all,
>
> the relationship and synergies of SMW and Wikidata are of course quite
> important to us, so I am taking the time to write a slightly longer
> email that I also send to the user list.
>
> At the moment, Wikidata is still under development, so it is a bit early
> to make concrete plans on how to align the project.
> Here are some things that I can already answer now:
>
>
> (1) What is the software in "Wikidata"?
>
> The software underlying Wikidata is called Wikibase. It consists of two
> extensions: Repository (where data is stored; this will run at
> wikidata.org) and Client (that pulls data into other MediaWiki sites;
> this will run at wikipedia.org). Both extensions will be installable on
> MediaWiki sites to get Wikidata-like features there.
>
>
> (2) What are the main differences between SMW and Wikidata?
>
> Wikidata is being built with SMW in mind, and the experiences with SMW
> are a valuable input for its design. The main use case of Wikidata (a
> centralised, multi-lingual site that serves as a data repository) is
> different from that of SMW (a data-enhanced MediaWiki), and this leads
> to a number of differences.
>
> The data model of Wikidata is a bit more complex than that of SMW. For
> example, every fact will also have an (optional) reference. Wikidata
> will also use more complex datatypes (e.g., since many text values can
> be translated). However, both Wikidata and SMW are still mostly based on
> property-value assignments, and Wikidata will have a Property namespace.
> Also, the more complex structures in Wikidata could be captured in SMW
> using internal objects.
>
> The user interface of Wikibase Repository will be based on input forms,
> and thus quite different from SMW. The UI will be a bit like a
> full-page, editable Factbox, but much more advanced, with a lot of
> support for entering values.
>
> It is not defined yet what kind of query language Wikidata will support
> in Phase 3 (or thereafter). This needs to be designed first, but the
> more complicated data model will also have an impact there.
>
>
> (3) What is a ContentHandler and how could it help SMW?
>
> ContentHandlers allow MediaWiki to manage textual content that is not
> interpreted as wikitext. Wikidata uses this to store data in a
> text-based format (JSON) without having to invent a MediaWiki syntax
> extension. In the future, it might be possible to have "multi-part"
> content for one page, so that one can store both Wikitext and other
> structured data in a cleaner way.
>
> SMW could use this to allow data to be stored directly for a page,
> without having to specify the data in the wikitext. This could be a bit
> like an "editable Factbox". MediaWiki could do something similar with
> Categories.
>
>
> (4) Will ContentHandlers reduce the dependence on custom SQL?
>
> No. Content that is stored in textual form on wiki pages is not suitable
> for answering queries quickly. To do this, it will always be necessary
> to have custom tables and/or external storage engines that provide fast
> access to the data. This is also the case for Wikidata.
>
> Currently, the MW page is the *primary* storage for SMW data, while the
> SQL tables are just *secondary* storage for faster lookup. The
> ContentHandlers can improve primary storage handling, but won't affect
> secondary storages and querying.
>
>
> (5) What are the possible synergies and joint uses of Wikidata and SMW?
>
> The details can only be worked out when Wikidata is a bit more stable
> than it is now, but there are a few obvious scenarios:
>
> * Wikibase Repository + SMW: data given in SMW annotations and data
> edited in Wikidata end up in the same secondary data structures, and can
> be retrieved through the same interfaces/queries. There are three main
> options: SMW tables are used for Wikibase content (this would yield
> query support via #ask), Wikibase tables are used for SMW content
> (yields full integration with Wikibase data access features), or both.
>
> * SMW as a Wikibase Repository: SMW could support the use of Wikibase
> APIs and exports on top of its native data store, so that other sites
> could fetch SMW data via the Wikidata Client.
>
> * SMW as a Wikibase Client: SMW could also pull data from Wikibase
> Repositories to use it locally and to display it in its interfaces.
>
> * Exchanging code: Besides application-level integration, SMW and
> Wikidata are also going to exchange lines of code. Initially, Wikidata
> will exploit code from SMW (e.g., the calendar conversion code for times
> and dates). Later, smaller pieces of Wikidata code might be used for
> SMW. In many cases, this will mean that the code is adapted heavily; if
> it can be used without much change, one would possibly create a "library
> extension" (like Validator).
>
>
> (6) Will Wikidata replace SMW in the long run?
>
> No. If anything, the integration between both could become so good that
> the distinction is no longer relevant and we are only talking about
> different data authoring and data access interface extensions to
> MediaWiki, which users can enable and disable at their leisure. This
> will still take some time though.
>
>
> (7) What is the development plan for SMW with respect to Wikidata?
>
> The strategy is to wait until Wikidata has settled a bit before
> approaching any integration. This will probably be another 10 or 11
> months from now (note that data management only comes in at Phase 2 of
> Wikidata, i.e., in 5 months). Exchanging some code might be possible
> before that, but integrating Wikibase while it is a moving target would
> not be feasible (there would not be a chance of keeping up with the
> speed of Wikidata, esp. since half the SMW team is engaged in Wikidata
> development right now).
>
>
> I hope this answers most questions on that topic for now. Maybe it would
> be nice to have a wikipage on semantic-mediawiki.org with these answers.
>
> Best regards,
>
> Markus
>
>
> On 30/04/12 23:44, James HK wrote:
>> Hi,
>>
>> ==WD ContentHandler ==
>> Now that the Wikidata project is introducing a new ContentHandler [1],
>> we where curious if in near future SMW would make use of those methods
>> to store its own objects in the same way and reduces its reliance on
>> non-mediawiki tables/methods.
>>
>> As Wikidata and SMW share the same bottlenecks in terms of
>> performance, caching, scalability it might prove beneficial for SMW to
>> use similar methods to store and fetch dataobjects from a mw-database
>> while at the same time reduces its reliance on SQL specific code and
>> lift the ban on other supported database systems(Oracle,Sqlite etc.).
>>
>> The reason why we are asking is that we hope that some of the work
>> that has be done in SMW is reflected in Wikidata as visa versa to
>> ensure that both developments share some similarities to the extend
>> that their "plug and playable"  (meaning causing the least
>> interruptions when installed together and make it easier for code
>> reviewers to understand the inner workings which increases stability).
>>
>> ==SMW and WD ==
>> Furthermore, with the Wikidata project on the horizon, we would be
>> interested to see how SMW positioned and distinguished itself from WD
>> in light of recent developments. (A simple table could help to explain
>> those differences fairly quick while at the same time allows some
>> directional guidance for the future of SMW.)
>>
>> ==Validator==
>> The Validator provides validation methods for datatypes (and parameter
>> definitions etc.) and WD certainly needs similar methods, is it
>> planned that those methods (or part of it) are being moved into the
>> core?
>>
>> [1] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki%2Fcore.git;a=commit;h=337f65c03b59eaddc038bc1a0fb150dc97e03154
>>
>> Cheers,
>>
>> mwjames
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Semediawiki-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Semediawiki-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: [SMW-devel] SMW and Wikidata (Was: Semantic MediaWiki and Wikidata ContentHandler)

Markus Krötzsch-2
On 01/05/12 11:45, Denny Vrandecic wrote:
> Markus,
>
> do you mind if we take this text as the starting point for a text detailing the relation between SMW and Wikidata here?

Not at all.

Markus

>
> <https://meta.wikimedia.org/wiki/Wikidata/Notes/SMW_and_Wikidata>
>
> Cheers,
> Denny
>
>
> On 1 May 2012, at 10:04, Markus Krötzsch wrote:
>
>> Hi James, hi all,
>>
>> the relationship and synergies of SMW and Wikidata are of course quite
>> important to us, so I am taking the time to write a slightly longer
>> email that I also send to the user list.
>>
>> At the moment, Wikidata is still under development, so it is a bit early
>> to make concrete plans on how to align the project.
>> Here are some things that I can already answer now:
>>
>>
>> (1) What is the software in "Wikidata"?
>>
>> The software underlying Wikidata is called Wikibase. It consists of two
>> extensions: Repository (where data is stored; this will run at
>> wikidata.org) and Client (that pulls data into other MediaWiki sites;
>> this will run at wikipedia.org). Both extensions will be installable on
>> MediaWiki sites to get Wikidata-like features there.
>>
>>
>> (2) What are the main differences between SMW and Wikidata?
>>
>> Wikidata is being built with SMW in mind, and the experiences with SMW
>> are a valuable input for its design. The main use case of Wikidata (a
>> centralised, multi-lingual site that serves as a data repository) is
>> different from that of SMW (a data-enhanced MediaWiki), and this leads
>> to a number of differences.
>>
>> The data model of Wikidata is a bit more complex than that of SMW. For
>> example, every fact will also have an (optional) reference. Wikidata
>> will also use more complex datatypes (e.g., since many text values can
>> be translated). However, both Wikidata and SMW are still mostly based on
>> property-value assignments, and Wikidata will have a Property namespace.
>> Also, the more complex structures in Wikidata could be captured in SMW
>> using internal objects.
>>
>> The user interface of Wikibase Repository will be based on input forms,
>> and thus quite different from SMW. The UI will be a bit like a
>> full-page, editable Factbox, but much more advanced, with a lot of
>> support for entering values.
>>
>> It is not defined yet what kind of query language Wikidata will support
>> in Phase 3 (or thereafter). This needs to be designed first, but the
>> more complicated data model will also have an impact there.
>>
>>
>> (3) What is a ContentHandler and how could it help SMW?
>>
>> ContentHandlers allow MediaWiki to manage textual content that is not
>> interpreted as wikitext. Wikidata uses this to store data in a
>> text-based format (JSON) without having to invent a MediaWiki syntax
>> extension. In the future, it might be possible to have "multi-part"
>> content for one page, so that one can store both Wikitext and other
>> structured data in a cleaner way.
>>
>> SMW could use this to allow data to be stored directly for a page,
>> without having to specify the data in the wikitext. This could be a bit
>> like an "editable Factbox". MediaWiki could do something similar with
>> Categories.
>>
>>
>> (4) Will ContentHandlers reduce the dependence on custom SQL?
>>
>> No. Content that is stored in textual form on wiki pages is not suitable
>> for answering queries quickly. To do this, it will always be necessary
>> to have custom tables and/or external storage engines that provide fast
>> access to the data. This is also the case for Wikidata.
>>
>> Currently, the MW page is the *primary* storage for SMW data, while the
>> SQL tables are just *secondary* storage for faster lookup. The
>> ContentHandlers can improve primary storage handling, but won't affect
>> secondary storages and querying.
>>
>>
>> (5) What are the possible synergies and joint uses of Wikidata and SMW?
>>
>> The details can only be worked out when Wikidata is a bit more stable
>> than it is now, but there are a few obvious scenarios:
>>
>> * Wikibase Repository + SMW: data given in SMW annotations and data
>> edited in Wikidata end up in the same secondary data structures, and can
>> be retrieved through the same interfaces/queries. There are three main
>> options: SMW tables are used for Wikibase content (this would yield
>> query support via #ask), Wikibase tables are used for SMW content
>> (yields full integration with Wikibase data access features), or both.
>>
>> * SMW as a Wikibase Repository: SMW could support the use of Wikibase
>> APIs and exports on top of its native data store, so that other sites
>> could fetch SMW data via the Wikidata Client.
>>
>> * SMW as a Wikibase Client: SMW could also pull data from Wikibase
>> Repositories to use it locally and to display it in its interfaces.
>>
>> * Exchanging code: Besides application-level integration, SMW and
>> Wikidata are also going to exchange lines of code. Initially, Wikidata
>> will exploit code from SMW (e.g., the calendar conversion code for times
>> and dates). Later, smaller pieces of Wikidata code might be used for
>> SMW. In many cases, this will mean that the code is adapted heavily; if
>> it can be used without much change, one would possibly create a "library
>> extension" (like Validator).
>>
>>
>> (6) Will Wikidata replace SMW in the long run?
>>
>> No. If anything, the integration between both could become so good that
>> the distinction is no longer relevant and we are only talking about
>> different data authoring and data access interface extensions to
>> MediaWiki, which users can enable and disable at their leisure. This
>> will still take some time though.
>>
>>
>> (7) What is the development plan for SMW with respect to Wikidata?
>>
>> The strategy is to wait until Wikidata has settled a bit before
>> approaching any integration. This will probably be another 10 or 11
>> months from now (note that data management only comes in at Phase 2 of
>> Wikidata, i.e., in 5 months). Exchanging some code might be possible
>> before that, but integrating Wikibase while it is a moving target would
>> not be feasible (there would not be a chance of keeping up with the
>> speed of Wikidata, esp. since half the SMW team is engaged in Wikidata
>> development right now).
>>
>>
>> I hope this answers most questions on that topic for now. Maybe it would
>> be nice to have a wikipage on semantic-mediawiki.org with these answers.
>>
>> Best regards,
>>
>> Markus
>>
>>
>> On 30/04/12 23:44, James HK wrote:
>>> Hi,
>>>
>>> ==WD ContentHandler ==
>>> Now that the Wikidata project is introducing a new ContentHandler [1],
>>> we where curious if in near future SMW would make use of those methods
>>> to store its own objects in the same way and reduces its reliance on
>>> non-mediawiki tables/methods.
>>>
>>> As Wikidata and SMW share the same bottlenecks in terms of
>>> performance, caching, scalability it might prove beneficial for SMW to
>>> use similar methods to store and fetch dataobjects from a mw-database
>>> while at the same time reduces its reliance on SQL specific code and
>>> lift the ban on other supported database systems(Oracle,Sqlite etc.).
>>>
>>> The reason why we are asking is that we hope that some of the work
>>> that has be done in SMW is reflected in Wikidata as visa versa to
>>> ensure that both developments share some similarities to the extend
>>> that their "plug and playable"  (meaning causing the least
>>> interruptions when installed together and make it easier for code
>>> reviewers to understand the inner workings which increases stability).
>>>
>>> ==SMW and WD ==
>>> Furthermore, with the Wikidata project on the horizon, we would be
>>> interested to see how SMW positioned and distinguished itself from WD
>>> in light of recent developments. (A simple table could help to explain
>>> those differences fairly quick while at the same time allows some
>>> directional guidance for the future of SMW.)
>>>
>>> ==Validator==
>>> The Validator provides validation methods for datatypes (and parameter
>>> definitions etc.) and WD certainly needs similar methods, is it
>>> planned that those methods (or part of it) are being moved into the
>>> core?
>>>
>>> [1] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki%2Fcore.git;a=commit;h=337f65c03b59eaddc038bc1a0fb150dc97e03154
>>>
>>> Cheers,
>>>
>>> mwjames
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Semediawiki-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Semediawiki-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: [SMW-devel] SMW and Wikidata (Was: Semantic MediaWiki and Wikidata ContentHandler)

kghbln
I think this information should also go onto http://semanticweb.org somehow. So far this [1] seems to be the
only related information which is a bit quaint in the meantime. ;) Cheers Karsten

[1] http://semanticweb.org/wiki/Semantic_Wikipedia

Am 01.05.2012 14:32, schrieb Markus Krötzsch:

> On 01/05/12 11:45, Denny Vrandecic wrote:
>> Markus,
>>
>> do you mind if we take this text as the starting point for a text detailing the relation between SMW and Wikidata here?
> Not at all.
>
> Markus
>
>> <https://meta.wikimedia.org/wiki/Wikidata/Notes/SMW_and_Wikidata>
>>
>> Cheers,
>> Denny
>>
>>
>> On 1 May 2012, at 10:04, Markus Krötzsch wrote:
>>
>>> Hi James, hi all,
>>>
>>> the relationship and synergies of SMW and Wikidata are of course quite
>>> important to us, so I am taking the time to write a slightly longer
>>> email that I also send to the user list.
>>>
>>> At the moment, Wikidata is still under development, so it is a bit early
>>> to make concrete plans on how to align the project.
>>> Here are some things that I can already answer now:
>>>
>>>
>>> (1) What is the software in "Wikidata"?
>>>
>>> The software underlying Wikidata is called Wikibase. It consists of two
>>> extensions: Repository (where data is stored; this will run at
>>> wikidata.org) and Client (that pulls data into other MediaWiki sites;
>>> this will run at wikipedia.org). Both extensions will be installable on
>>> MediaWiki sites to get Wikidata-like features there.
>>>
>>>
>>> (2) What are the main differences between SMW and Wikidata?
>>>
>>> Wikidata is being built with SMW in mind, and the experiences with SMW
>>> are a valuable input for its design. The main use case of Wikidata (a
>>> centralised, multi-lingual site that serves as a data repository) is
>>> different from that of SMW (a data-enhanced MediaWiki), and this leads
>>> to a number of differences.
>>>
>>> The data model of Wikidata is a bit more complex than that of SMW. For
>>> example, every fact will also have an (optional) reference. Wikidata
>>> will also use more complex datatypes (e.g., since many text values can
>>> be translated). However, both Wikidata and SMW are still mostly based on
>>> property-value assignments, and Wikidata will have a Property namespace.
>>> Also, the more complex structures in Wikidata could be captured in SMW
>>> using internal objects.
>>>
>>> The user interface of Wikibase Repository will be based on input forms,
>>> and thus quite different from SMW. The UI will be a bit like a
>>> full-page, editable Factbox, but much more advanced, with a lot of
>>> support for entering values.
>>>
>>> It is not defined yet what kind of query language Wikidata will support
>>> in Phase 3 (or thereafter). This needs to be designed first, but the
>>> more complicated data model will also have an impact there.
>>>
>>>
>>> (3) What is a ContentHandler and how could it help SMW?
>>>
>>> ContentHandlers allow MediaWiki to manage textual content that is not
>>> interpreted as wikitext. Wikidata uses this to store data in a
>>> text-based format (JSON) without having to invent a MediaWiki syntax
>>> extension. In the future, it might be possible to have "multi-part"
>>> content for one page, so that one can store both Wikitext and other
>>> structured data in a cleaner way.
>>>
>>> SMW could use this to allow data to be stored directly for a page,
>>> without having to specify the data in the wikitext. This could be a bit
>>> like an "editable Factbox". MediaWiki could do something similar with
>>> Categories.
>>>
>>>
>>> (4) Will ContentHandlers reduce the dependence on custom SQL?
>>>
>>> No. Content that is stored in textual form on wiki pages is not suitable
>>> for answering queries quickly. To do this, it will always be necessary
>>> to have custom tables and/or external storage engines that provide fast
>>> access to the data. This is also the case for Wikidata.
>>>
>>> Currently, the MW page is the *primary* storage for SMW data, while the
>>> SQL tables are just *secondary* storage for faster lookup. The
>>> ContentHandlers can improve primary storage handling, but won't affect
>>> secondary storages and querying.
>>>
>>>
>>> (5) What are the possible synergies and joint uses of Wikidata and SMW?
>>>
>>> The details can only be worked out when Wikidata is a bit more stable
>>> than it is now, but there are a few obvious scenarios:
>>>
>>> * Wikibase Repository + SMW: data given in SMW annotations and data
>>> edited in Wikidata end up in the same secondary data structures, and can
>>> be retrieved through the same interfaces/queries. There are three main
>>> options: SMW tables are used for Wikibase content (this would yield
>>> query support via #ask), Wikibase tables are used for SMW content
>>> (yields full integration with Wikibase data access features), or both.
>>>
>>> * SMW as a Wikibase Repository: SMW could support the use of Wikibase
>>> APIs and exports on top of its native data store, so that other sites
>>> could fetch SMW data via the Wikidata Client.
>>>
>>> * SMW as a Wikibase Client: SMW could also pull data from Wikibase
>>> Repositories to use it locally and to display it in its interfaces.
>>>
>>> * Exchanging code: Besides application-level integration, SMW and
>>> Wikidata are also going to exchange lines of code. Initially, Wikidata
>>> will exploit code from SMW (e.g., the calendar conversion code for times
>>> and dates). Later, smaller pieces of Wikidata code might be used for
>>> SMW. In many cases, this will mean that the code is adapted heavily; if
>>> it can be used without much change, one would possibly create a "library
>>> extension" (like Validator).
>>>
>>>
>>> (6) Will Wikidata replace SMW in the long run?
>>>
>>> No. If anything, the integration between both could become so good that
>>> the distinction is no longer relevant and we are only talking about
>>> different data authoring and data access interface extensions to
>>> MediaWiki, which users can enable and disable at their leisure. This
>>> will still take some time though.
>>>
>>>
>>> (7) What is the development plan for SMW with respect to Wikidata?
>>>
>>> The strategy is to wait until Wikidata has settled a bit before
>>> approaching any integration. This will probably be another 10 or 11
>>> months from now (note that data management only comes in at Phase 2 of
>>> Wikidata, i.e., in 5 months). Exchanging some code might be possible
>>> before that, but integrating Wikibase while it is a moving target would
>>> not be feasible (there would not be a chance of keeping up with the
>>> speed of Wikidata, esp. since half the SMW team is engaged in Wikidata
>>> development right now).
>>>
>>>
>>> I hope this answers most questions on that topic for now. Maybe it would
>>> be nice to have a wikipage on semantic-mediawiki.org with these answers.
>>>
>>> Best regards,
>>>
>>> Markus
>>>
>>>
>>> On 30/04/12 23:44, James HK wrote:
>>>> Hi,
>>>>
>>>> ==WD ContentHandler ==
>>>> Now that the Wikidata project is introducing a new ContentHandler [1],
>>>> we where curious if in near future SMW would make use of those methods
>>>> to store its own objects in the same way and reduces its reliance on
>>>> non-mediawiki tables/methods.
>>>>
>>>> As Wikidata and SMW share the same bottlenecks in terms of
>>>> performance, caching, scalability it might prove beneficial for SMW to
>>>> use similar methods to store and fetch dataobjects from a mw-database
>>>> while at the same time reduces its reliance on SQL specific code and
>>>> lift the ban on other supported database systems(Oracle,Sqlite etc.).
>>>>
>>>> The reason why we are asking is that we hope that some of the work
>>>> that has be done in SMW is reflected in Wikidata as visa versa to
>>>> ensure that both developments share some similarities to the extend
>>>> that their "plug and playable"  (meaning causing the least
>>>> interruptions when installed together and make it easier for code
>>>> reviewers to understand the inner workings which increases stability).
>>>>
>>>> ==SMW and WD ==
>>>> Furthermore, with the Wikidata project on the horizon, we would be
>>>> interested to see how SMW positioned and distinguished itself from WD
>>>> in light of recent developments. (A simple table could help to explain
>>>> those differences fairly quick while at the same time allows some
>>>> directional guidance for the future of SMW.)
>>>>
>>>> ==Validator==
>>>> The Validator provides validation methods for datatypes (and parameter
>>>> definitions etc.) and WD certainly needs similar methods, is it
>>>> planned that those methods (or part of it) are being moved into the
>>>> core?
>>>>
>>>> [1] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki%2Fcore.git;a=commit;h=337f65c03b59eaddc038bc1a0fb150dc97e03154
>>>>
>>>> Cheers,
>>>>
>>>> mwjames
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond. Discussions
>>>> will include endpoint security, mobile security and the latest in malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Semediawiki-devel mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Semediawiki-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Semediawiki-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user