speed of SMW_refreshData.php

[Benjamin Good-3]

I am working on upgrading a fairly large smw install (75,000 pages) to the
new SMWSQLStore3.  I am on the step
php SMW_refreshData.php -v -b SMWSQLStore3
from here
http://semantic-mediawiki.org/wiki/Help:Installation#With_shell_access

It seems to be working, but it is extremely slow.  Estimating its going to
take 3-4 days to finish on a 2.3 gh macbook.
Is there any way to speed it up ?

thanks!
-Ben
------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[thingles]

If you specify start and end pages you could run more than one at a time. I would think it would be fine to run 1 per core in your machine without too much issue. You could write a bash script to do this or just to it by hand with a few prompts open.
Jamie Thingelstad
[hidden email]
mobile: [phone]
find me on AIM Twitter Facebook LinkedIn

On Apr 15, 2013, at 7:15 PM, Benjamin Good <[hidden email]> wrote:

> I am working on upgrading a fairly large smw install (75,000 pages) to the
> new SMWSQLStore3.  I am on the step
> php SMW_refreshData.php -v -b SMWSQLStore3
> from here
> http://semantic-mediawiki.org/wiki/Help:Installation#With_shell_access
>
> It seems to be working, but it is extremely slow.  Estimating its going to
> take 3-4 days to finish on a 2.3 gh macbook.
> Is there any way to speed it up ?
>
> thanks!
> -Ben
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Benjamin Good-3]

Thanks Jamie!,

Not knowing much about whats going on behind the scenes I wouldn't have
tried to get things going in parallel..  Three processes are running
simultaneously now without any issue and it seems I could probably get a
few more going.  Much faster!

yay
-Ben


On Mon, Apr 15, 2013 at 6:00 PM, Jamie Thingelstad <[hidden email]>wrote:

> If you specify start and end pages you could run more than one at a time.
> I would think it would be fine to run 1 per core in your machine without
> too much issue. You could write a bash script to do this or just to it by
> hand with a few prompts open.
>  Jamie Thingelstad <http://www.thingelstad.com/>
> [hidden email]
> mobile: [phone]
> find me on AIM Twitter <http://twitter.com/thingles> Facebook<http://www.facebook.com/thingles>
> LinkedIn <http://www.linkedin.com/in/jthingelstad>
>
> On Apr 15, 2013, at 7:15 PM, Benjamin Good <[hidden email]>
> wrote:
>
> I am working on upgrading a fairly large smw install (75,000 pages) to the
> new SMWSQLStore3.  I am on the step
> php SMW_refreshData.php -v -b SMWSQLStore3
> from here
> http://semantic-mediawiki.org/wiki/Help:Installation#With_shell_access
>
> It seems to be working, but it is extremely slow.  Estimating its going to
> take 3-4 days to finish on a 2.3 gh macbook.
> Is there any way to speed it up ?
>
> thanks!
> -Ben
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Semediawiki-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>
>
>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user