SMW 1.9 - Installation Concerns and Questions

[Ralf Krüdewagen]

Hi,

first of all I would like to thank all of you who have contributed to this new
release.

But to be honest I have some problems and concerns about the installation and
upgrade procedure.

1. Concerns

1.1 I think that composer is - at least for me as a non-developer but advanced
Linux and MW/SMW admin/user - a step back. Why? Because composer is an
additional layer on top of my Linux package management and git. And it's also
about trust (what is composer doing on my machine?)

I want to control which version of each extension and library is installed on
my machine. This is greatly supported using yum/zypper/rpm and git. But with
SMW 1.9 it seems to be almost impossible to just use git do fulfill all the
requirements. That's bad for me.

And in addition: On at least one of my Linux boxes there is AFAIK no PHP PHAR
package available. So, composer will probably not work and I must rely on the
ZIP-Bundle or on git (which I would prefer).

1.2 The Validator extension in the ZIP-Archiv seems to be bundled now with SMW
under "SemanticMediaWiki/extensions/Validator/".

It's unclear for me how that works with other extensions like SideBarMenu. And
what about the other requirements like ParamProcessor. Even though I was able
to clone the git repo, the docs doesn't explain well enough how to add
ParamProcessor to Validator.

1.3 So, I wish would be to have also an "manual" install doc of put all
components together via git.

2. Questions

2.1 If I want to install the ZIP-Bundle of SMW 1.9, the update doc says:

Remove
    require_once "$IP/extensions/Validator/Validator.php";


Well, this is - I guess - since Validator is now bundled with SMW under
"SemanticMediaWiki/extensions/Validator/".

But: How about other extensions using Validator like SideBarMenu (which I have
in use)? Can I run a 0.5.1 version of Validator installed in the regular MW
extensions dir in parallel with a 1.0.x. version installed in the SMW
extension?

Again, SMW is really great, but I was not able to update my site from 1.8.x to
1.9 within 1,5 hours (avoiding the composer stuff).

Any suggestions? How to use just git to bring all SMW components together?

Cheers
Ralf


------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Jeroen De Dauw-2]

Hey,

1.3 So, I wish would be to have also an "manual" install doc of put all
> components together via git.
>

This is simply no longer supported. We are using PHP libraries that you
cannot simply load doing a require_once, as they define how their
autoloading works in a declarative style.

But: How about other extensions using Validator like SideBarMenu (which I
> have
> in use)? Can I run a 0.5.1 version of Validator installed in the regular MW
> extensions dir in parallel with a 1.0.x. version installed in the SMW
> extension?
>

If SideBarMenu uses Validator and does not provide a version that supports
Composer yet, then you are out of luck and cannot upgrade to SMW 1.9 yet.
Once it supports Composer, you will be able to install both extensions via
Composer. Or you could find someone to build a tarball containing both for
you, though I'm not sure what good that would do. Note how our tarballs are
build using Composer [0], and how we make such a build on every change made
to our repo [1].

[0]
https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/build/release/build_tarballs.sh
[1]
https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/build/travis/before_script.sh#L59

I want to control which version of each extension and library is installed
> on
> my machine
>

You can see what exactly got installed in composer.lock. You can then
modify composer.json to include all dependencies of SMW and specify the
version ranges you require there. Of course in whatever way this is
achieved, you will end up with your box having knowledge of SMWs
dependencies, which is something generally should be avoided. Still, if you
feel this trade-off is worth it for you, explicitly adding the dependencies
in composer.json is the way to go.

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil. ~=[,,_,,]:3
--
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Ralf Krüdewagen]

Hi Jeroen,

thanks for your feedback.

Then I in deed need to understand and learn more in detail how composer works,
if composer is the only way to run SMW in the future (either by manually
installing or by the ZIP-Archive which is obviously made using composer).

I have one question about the interference with the extensions in the global
dir "<mw-install-dir>/extensions/" and the extensions (libraries) installed
under each project like in "SemanticMediaWiki/extensions/".

Even Semantic Result Formats seems not to have a "composer-driven"
installation method yet, but it uses Validator too. So, Validator still needs
to be installed in "<mw-install-dir>/extensions/Validator" for SRF, right?

And I hope that SMW does still use the library in
"SemanticMediaWiki/extensions/Validator/", right?

And this will work, right?
Or do I also need to have a composer-installed SRF extension?

Cheers
Ralf
PS: Another side effect of using composer: the community will not be able to
help as quick as before testing new features or bug fixes, since a simple "git
pull" will not work to activate a new commit.
 

On Tuesday 07 January 2014 Jeroen De Dauw wrote:

> Hey,
>
> 1.3 So, I wish would be to have also an "manual" install doc of put all
>
> > components together via git.
>
> This is simply no longer supported. We are using PHP libraries that you
> cannot simply load doing a require_once, as they define how their
> autoloading works in a declarative style.
>
> But: How about other extensions using Validator like SideBarMenu (which I
>
> > have
> > in use)? Can I run a 0.5.1 version of Validator installed in the regular
> > MW
> > extensions dir in parallel with a 1.0.x. version installed in the SMW
> > extension?
>
> If SideBarMenu uses Validator and does not provide a version that supports
> Composer yet, then you are out of luck and cannot upgrade to SMW 1.9 yet.
> Once it supports Composer, you will be able to install both extensions via
> Composer. Or you could find someone to build a tarball containing both for
> you, though I'm not sure what good that would do. Note how our tarballs are
> build using Composer [0], and how we make such a build on every change made
> to our repo [1].
>
> [0]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/build/rel
> ease/build_tarballs.sh [1]
> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/build/tra
> vis/before_script.sh#L59
>
> I want to control which version of each extension and library is installed
>
> > on
> > my machine
>
> You can see what exactly got installed in composer.lock. You can then
> modify composer.json to include all dependencies of SMW and specify the
> version ranges you require there. Of course in whatever way this is
> achieved, you will end up with your box having knowledge of SMWs
> dependencies, which is something generally should be avoided. Still, if you
> feel this trade-off is worth it for you, explicitly adding the dependencies
> in composer.json is the way to go.
>
> Cheers
>
> --
> Jeroen De Dauw
> http://www.bn2vs.com
> Don't panic. Don't be evil. ~=[,,_,,]:3
> --

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Jeroen De Dauw-2]

Hey,

I have one question about the interference with the extensions in the global
> dir "<mw-install-dir>/extensions/" and the extensions (libraries) installed
> under each project like in "SemanticMediaWiki/extensions/".
>

If you use our recommended installation approach, all components end up in
either "<mw-install-dir>/extensions" or "<mw-install-dir>/vendor". If you
use the tarball, then those directories will be under the SemanticMediaWiki
directory, and then you will not be able to use Composer to install other
extensions without creating problems.

Even Semantic Result Formats seems not to have a "composer-driven"
> installation method yet, but it uses Validator too. So, Validator still
> needs
> to be installed in "<mw-install-dir>/extensions/
> Validator" for SRF, right?
>

This is incorrect. SRF uses the same process as SMW. You can install SRF
1.9 alpha and SMW 1.9 together quite easily. This is what is currently done
on the SMW wiki. See https://semantic-mediawiki.org/wiki/Special:Version

PS: Another side effect of using composer: the community will not be able to
> help as quick as before testing new features or bug fixes, since a simple
> "git
> pull" will not work to activate a new commit.
>

One can still run the development version. Combine that with releases being
more easily to make, and updating ones extensions being a lot simpler, I
anticipate people to run much more recent versions of the software than has
been the case historically.

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil. ~=[,,_,,]:3
--
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Ralf Krüdewagen]

Hey,

okay, then it seems that I need to wait for a SMW 1.9 update until all
Semantic family extensions are SMW 1.9 proven and the composer way fits to my
private and company MW installations at all.

Since
  ... I am running SRF 1.8, which seems not to work with composer-installed
Validator + ParamProcessor + SMW 1.9
  ... I can't find SRF 1.9 alpha
  ... I am running other extensions (SideBarMenu) depending on Validator but
are not composer-installed
  ... I am currently running Validator 0.5.1 without ParamProcessor
  ... but Validator 1.x is needed which needs ParamProcessor which only works
using composer (or by hacking the code to use the lib)

I am confused and need to sleep a night about it :-) Don't worry ;-) ...

Cheers
Ralf

On Tuesday 07 January 2014 Jeroen De Dauw wrote:

> Hey,
>
> I have one question about the interference with the extensions in the global
> > dir "<mw-install-dir>/extensions/" and the extensions (libraries)
> > installed
> > under each project like in "SemanticMediaWiki/extensions/".
>
> If you use our recommended installation approach, all components end up in
> either "<mw-install-dir>/extensions" or "<mw-install-dir>/vendor". If you
> use the tarball, then those directories will be under the SemanticMediaWiki
> directory, and then you will not be able to use Composer to install other
> extensions without creating problems.
>
> Even Semantic Result Formats seems not to have a "composer-driven"
>
> > installation method yet, but it uses Validator too. So, Validator still
> > needs
> > to be installed in "<mw-install-dir>/extensions/
> > Validator" for SRF, right?
>
> This is incorrect. SRF uses the same process as SMW. You can install SRF
> 1.9 alpha and SMW 1.9 together quite easily. This is what is currently done
> on the SMW wiki. See https://semantic-mediawiki.org/wiki/Special:Version
>
> PS: Another side effect of using composer: the community will not be able to
> > help as quick as before testing new features or bug fixes, since a simple
> > "git
> > pull" will not work to activate a new commit.
>
> One can still run the development version. Combine that with releases being
> more easily to make, and updating ones extensions being a lot simpler, I
> anticipate people to run much more recent versions of the software than has
> been the case historically.
>
> Cheers
>
> --
> Jeroen De Dauw
> http://www.bn2vs.com
> Don't panic. Don't be evil. ~=[,,_,,]:3
> --

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Eric_]

Jeroen De Dauw wrote

If you use our recommended installation approach, all components end up in
either "<mw-install-dir>/extensions" or "<mw-install-dir>/vendor". If you
use the tarball, then those directories will be under the SemanticMediaWiki
directory, and then you will not be able to use Composer to install other
extensions without creating problems.

Considering you've already upgraded to 1.9.0.1 using the tarball what would be the recommended way to switch to the composer method?

[Jeroen De Dauw-2]

Hey,

Considering you've already upgraded to 1.9.0.1 using the tarball what would
> be the recommended way to switch to the composer method?
>

Remove extensions/SemanticMediaWiki and then run the composer require
command listed in the installation instructions.

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil. ~=[,,_,,]:3
--
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Eric_]

Sorry for flooding with questions but I'm stuck...

It stops with this message:

[...@... w]# php composer.phar require mediawiki/semantic-media-wiki "1.9.*,>=1.9.0.1"
./composer.json has been updated
Loading composer repositories with package information



  [Symfony\Component\Process\Exception\ProcessTimedOutException]
  The process "git clone --mirror 'https://gerrit.wikimedia.org/r/p/mediawiki/core.git' '/root/.composer/cache/vcs/https---gerrit.wikimedia.org-r-p-mediawiki-core.git/'" exceeded the time
  out of 300 seconds.



require [--dev] [--prefer-source] [--prefer-dist] [--no-progress] [--no-update] [packages1] ... [packagesN]

Any ideas? I guess it isn't supposed to take longer than 300 seconds.

Tyvm in advance.

[Jeroen De Dauw-2]

Hey,

Apparently it is trying to clone MediaWiki... it should not do this. Did
you somehow specify it should install MediaWiki as well? As you are
installing things _into_ MediaWiki, you should definitely not try to
install MediaWiki as well. Note how there also is no working MediaWiki
package out there, only experimental ones that are quite broken.

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil. ~=[,,_,,]:3
--
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Eric_]

Jeroen De Dauw-2 wrote

Apparently it is trying to clone MediaWiki... it should not do this. Did
you somehow specify it should install MediaWiki as well? As you are
installing things _into_ MediaWiki, you should definitely not try to
install MediaWiki as well. Note how there also is no working MediaWiki
package out there, only experimental ones that are quite broken.

The bigger picture:


* upgraded to MW 1.22 last month

* upgraded to SMW 1.9.0.1 using the tarball

* realized Maps 2.x doesn't work with Validator 1.x and Maps 3.x requires composer

<lot of hassle finally getting shell access and finding out how everything works>

* installed composer

* trying to upgrade Maps with "php composer.phar require mediawiki/maps "*"" gave:

./composer.json has been updated
Loading composer repositories with package information

  [RuntimeException]
  Failed to clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git, git was not found, check that it is installed and in your PATH env.

  sh: git: command not found

require [--dev] [--prefer-source] [--prefer-dist] [--no-progress] [--no-update] [packages1] ... [packagesN]

* installed git

* retrying with "php composer.phar require mediawiki/maps "*"" gave:

./composer.json has been updated
Loading composer repositories with package information

  [Symfony\Component\Process\Exception\ProcessTimedOutException]
  The process "git clone --mirror 'https://gerrit.wikimedia.org/r/p/mediawiki/core.git' '/root/.composer/cache/vcs/https---gerrit.wikimedia.org-r-p-mediawiki-core.git/'" exceeded the time
  out of 300 seconds.

require [--dev] [--prefer-source] [--prefer-dist] [--no-progress] [--no-update] [packages1] ... [packagesN]

 
* read your comment regarding potential difficulties from installing extensions using composer while SMW has been installed using the tarball

* trying to switch SMW from tarball to composer with "php composer.phar require mediawiki/semantic-media-wiki "1.9.*,>=1.9.0.1"" resulted in the same message as further above:

./composer.json has been updated
Loading composer repositories with package information

  [Symfony\Component\Process\Exception\ProcessTimedOutException]
  The process "git clone --mirror 'https://gerrit.wikimedia.org/r/p/mediawiki/core.git' '/root/.composer/cache/vcs/https---gerrit.wikimedia.org-r-p-mediawiki-core.git/'" exceeded the time
  out of 300 seconds.

require [--dev] [--prefer-source] [--prefer-dist] [--no-progress] [--no-update] [packages1] ... [packagesN]

The require part in composer.json currently looks like this:

        "require": {
                "php": ">=5.3.2",
                "mediawiki/semantic-media-wiki": "1.9.*,>=1.9.0.1",
                "mediawiki/maps": "*"
        },

During the whole process I've been following installation instructions and recommendations thoroughly. I'm not aware that in the process any cloning of MW was triggered and if so it wasn't intended.

So maybe next I would try without the ""mediawiki/maps": "*"" line in composer.jon but I really don't know what I'm actually doing, tyvm for bearing with me.

[Jeroen De Dauw-2]

Hey Eric,

Sorry for all the hassle you are running into!

Given the steps you described, I'm still at a loss why Composer would try
to clone MediaWiki. One thing I noticed is that you had to install git.
That is silly and should not be the case. Composer is capable of fetching
the code as tarballs. I don't have any machine that does not have git on
it, though suspect that git will not be used if the require/install/update
commands are run with "--prefer-dist --no-dev". It'd be great if someone
could confirm this so we can update the docs.

>       "require": {
>               "php": ">=5.3.2",
>               "mediawiki/semantic-media-wiki": "1.9.*,>=1.9.0.1",
>               "mediawiki/maps": "*"
>       },

Beware that the maps package has not had any stable release yet. So unless
you are specifying the minimum-stability [0], Composer will tell you there
is no such package. You can also simply use the following to specify you
want to use the development version of Maps:

"mediawiki/maps": "@dev"

[0] http://getcomposer.org/doc/04-schema.md#minimum-stability

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil. ~=[,,_,,]:3
--
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Semediawiki-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

[Eric_]

Hola,

I've managed to track down the root of the composer/install issues. It was an existing composer.json file from MW 1.22 RC0 that survived the upgrade to MW 1.22 stable. It's btw still included in current 1.21 branches.

With that file removed everything went smooth, incl. the upgrade from SRF 1.8 to SRF 1.9.

So it's now proven that even people with little to no idea can switch to composer. And after the SRF upgrade I can say that upgrading extensions has never been that easy.