When is it required to run rebuildData.php

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

When is it required to run rebuildData.php

James Montalvo
When is it required to run rebuildData.php? I haven't seen clear
explanation anywhere, but I apologize if I just missed it. Specifically I'm
wondering if it's required, recommended, or not recommended when upgrading
from MW1.25+SMW2.4 to MW1.27+SMW2.5

Thanks,
James
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: When is it required to run rebuildData.php

kghbln
Heiya James,

in general I do a full data rebuild every time I am required to run a
database update due to schema changes though this is not a hard
requirement. However I'd recommend doing so. I believe the last time
this was really required during upgrade was when changing from
SMWSQLStore2 to SMWSQLStore3.

There are however other occasions when you do indeed have to do a full
data rebuild, e.g. if you add or remove page related special properties,
special properties in connection with query profiling or fixed
properties, etc. These required runs are basically connected changes to
the settings of configuration parameters. The documentation pages on
these configuration parameters will state the requirement to rebuild the
data.

Moreover I rebuild the data on a regular basis as part of normal wiki
hygiene, e.g. to clear outdated entities. Here I use the
--shallow-update flag to the script. The frequency depends on the level
of activity happening on your wiki but once a week is probably a good
recommendation.

Cheers Karsten


Am 24.04.2017 um 19:59 schrieb James Montalvo:

> When is it required to run rebuildData.php? I haven't seen clear
> explanation anywhere, but I apologize if I just missed it. Specifically I'm
> wondering if it's required, recommended, or not recommended when upgrading
> from MW1.25+SMW2.4 to MW1.27+SMW2.5
>
> Thanks,
> James
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: When is it required to run rebuildData.php

James Montalvo
Karsten,

Thanks so much for the info. So would you agree that when moving from
MW1.25+SMW2.4 to MW1.27+SMW2.5 we should run rebuildData.php, but it is not
time critical. In other words, the site will still be functional, and if it
takes us a day to complete rebuildData.php it won't be a major problem?
We're moving to a new server, and importing the data takes about 2 hours,
building the elasticsearch index takes about 3, and SMW data rebuild takes
about 12, so I'd like to make the SMW data rebuild last.

Thanks again,
James

On Mon, Apr 24, 2017 at 2:03 PM, [[kgh]] <[hidden email]> wrote:

> Heiya James,
>
> in general I do a full data rebuild every time I am required to run a
> database update due to schema changes though this is not a hard
> requirement. However I'd recommend doing so. I believe the last time
> this was really required during upgrade was when changing from
> SMWSQLStore2 to SMWSQLStore3.
>
> There are however other occasions when you do indeed have to do a full
> data rebuild, e.g. if you add or remove page related special properties,
> special properties in connection with query profiling or fixed
> properties, etc. These required runs are basically connected changes to
> the settings of configuration parameters. The documentation pages on
> these configuration parameters will state the requirement to rebuild the
> data.
>
> Moreover I rebuild the data on a regular basis as part of normal wiki
> hygiene, e.g. to clear outdated entities. Here I use the
> --shallow-update flag to the script. The frequency depends on the level
> of activity happening on your wiki but once a week is probably a good
> recommendation.
>
> Cheers Karsten
>
>
> Am 24.04.2017 um 19:59 schrieb James Montalvo:
> > When is it required to run rebuildData.php? I haven't seen clear
> > explanation anywhere, but I apologize if I just missed it. Specifically
> I'm
> > wondering if it's required, recommended, or not recommended when
> upgrading
> > from MW1.25+SMW2.4 to MW1.27+SMW2.5
> >
> > Thanks,
> > James
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Semediawiki-user mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/semediawiki-user
> >
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user
Reply | Threaded
Open this post in threaded view
|

Re: When is it required to run rebuildData.php

James HK
Hi,

> Thanks so much for the info. So would you agree that when moving from
> MW1.25+SMW2.4 to MW1.27+SMW2.5 we should run rebuildData.php, but it is not
> time critical. In other words, the site will still be functional, and if it

For 2.5 it is not necessary and your wiki will be functional but
running the `rebuildData.php` will ensure that outdated entities [0]
are removed and stricter naming rules can be applied [1] without much
delay.

`update.php` is required to run as noted in [2].

> building the elasticsearch index takes about 3, and SMW data rebuild takes
> about 12, so I'd like to make the SMW data rebuild last.

When building an index from tables as done so by elasticsearch or our
own [3] the system can directly access the table content and process
them without further requirements to run other MediaWiki functions
(incl. the Parser).

Unfortunately, we cannot do this with the `rebuildData.php` script and
are constrained to go through the complete Parser cycle to ensure that
all data (incl. modification date, category links, extension modified
data etc.) are available when updating the `SemanticData` object.

[0] https://www.semantic-mediawiki.org/wiki/Help:Outdated_entities
[1] https://www.semantic-mediawiki.org/wiki/Help:$smwgPropertyInvalidCharacterList
[2] https://www.semantic-mediawiki.org/wiki/Semantic_MediaWiki_2.5.1
[3] https://www.semantic-mediawiki.org/wiki/Help:Full-text_search/Indexing

Cheers

On 4/25/17, James Montalvo <[hidden email]> wrote:

> Karsten,
>
> Thanks so much for the info. So would you agree that when moving from
> MW1.25+SMW2.4 to MW1.27+SMW2.5 we should run rebuildData.php, but it is not
> time critical. In other words, the site will still be functional, and if it
> takes us a day to complete rebuildData.php it won't be a major problem?
> We're moving to a new server, and importing the data takes about 2 hours,
> building the elasticsearch index takes about 3, and SMW data rebuild takes
> about 12, so I'd like to make the SMW data rebuild last.
>
> Thanks again,
> James
>
> On Mon, Apr 24, 2017 at 2:03 PM, [[kgh]] <[hidden email]> wrote:
>
>> Heiya James,
>>
>> in general I do a full data rebuild every time I am required to run a
>> database update due to schema changes though this is not a hard
>> requirement. However I'd recommend doing so. I believe the last time
>> this was really required during upgrade was when changing from
>> SMWSQLStore2 to SMWSQLStore3.
>>
>> There are however other occasions when you do indeed have to do a full
>> data rebuild, e.g. if you add or remove page related special properties,
>> special properties in connection with query profiling or fixed
>> properties, etc. These required runs are basically connected changes to
>> the settings of configuration parameters. The documentation pages on
>> these configuration parameters will state the requirement to rebuild the
>> data.
>>
>> Moreover I rebuild the data on a regular basis as part of normal wiki
>> hygiene, e.g. to clear outdated entities. Here I use the
>> --shallow-update flag to the script. The frequency depends on the level
>> of activity happening on your wiki but once a week is probably a good
>> recommendation.
>>
>> Cheers Karsten
>>
>>
>> Am 24.04.2017 um 19:59 schrieb James Montalvo:
>> > When is it required to run rebuildData.php? I haven't seen clear
>> > explanation anywhere, but I apologize if I just missed it. Specifically
>> I'm
>> > wondering if it's required, recommended, or not recommended when
>> upgrading
>> > from MW1.25+SMW2.4 to MW1.27+SMW2.5
>> >
>> > Thanks,
>> > James
>> > ------------------------------------------------------------
>> ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Semediawiki-user mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>> >
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Semediawiki-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user