Sigh, bulk image addition--so near and yet so far...

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Sigh, bulk image addition--so near and yet so far...

Ken McDonald-2
With many thanks to responses to previous questions, I think I _almost_
at the point of being able to bulk upload images. Here's the situation:

  - My images are in a file "uploads" in the "main" wiki directory, i.e.
the one containing the "maintenance" directory. They all have a .jpg suffix
  - I have added and configured an "AdminSettings.php" file in the main
directory. I can now run the rebuildImages script, which I could not do
before.

I get two different types of results when I try to execute the
rebuildImages script while in the maintenance directory:

If I run "php rebuildImages.php", I get the result;

wikidb 2006-05-17 18:43:07: Processing image...
wikidb 2006-05-17 18:43:07: Finished image... 0 of 2 rows updated
wikidb 2006-05-17 18:43:07: Processing oldimage...
wikidb 2006-05-17 18:43:07: Finished oldimage... 0 of 0 rows updated

I have two images already in the wiki, I assume that is what is being
reflected in this output.


If I run "php rebuildImages.php --missing", The result is much longer,
starting with:

wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/00
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/01
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/02
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/03
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/04
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/05
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/06
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/07
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/08
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/09
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/0a
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/0b
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/0c
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/0d
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/0e
wikidb 2006-05-17 18:45:00: no directory, skipping
/Users/Ken/Sites/mediawiki-1.6.5/images/0/0f

and so on...


I'm assuming this causes rebuildImages to look through the "images" dir,
looking for files it doesn't know about, and then add them to the database.

Does this mean I need to add my files to that directory before running
the script? If so, where? The directory apparently has a very specific
structure, and I don't want to put images in incorrect places. Or can I
get "rebuildImages" to look at another directory (eg.
/wikimain/uploads), move them into the appropriate place in "images",
and register the with the database?

Once again, so many thanks for helping with this. We have (at least)
several hundred images that need to be put into the site, and don't
really like the thought of doing it by hand :-).
_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Rob Church
On 17/05/06, Ken McDonald <[hidden email]> wrote:

> With many thanks to responses to previous questions, I think I _almost_
> at the point of being able to bulk upload images. Here's the situation:
>
>   - My images are in a file "uploads" in the "main" wiki directory, i.e.
> the one containing the "maintenance" directory. They all have a .jpg suffix
>   - I have added and configured an "AdminSettings.php" file in the main
> directory. I can now run the rebuildImages script, which I could not do
> before.
>
> I get two different types of results when I try to execute the
> rebuildImages script while in the maintenance directory:
>
> If I run "php rebuildImages.php", I get the result;
>
> wikidb 2006-05-17 18:43:07: Processing image...
> wikidb 2006-05-17 18:43:07: Finished image... 0 of 2 rows updated
> wikidb 2006-05-17 18:43:07: Processing oldimage...
> wikidb 2006-05-17 18:43:07: Finished oldimage... 0 of 0 rows updated
>
> I have two images already in the wiki, I assume that is what is being
> reflected in this output.
>
>
> If I run "php rebuildImages.php --missing", The result is much longer,
> starting with:
>
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/00
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/01
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/02
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/03
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/04
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/05
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/06
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/07
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/08
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/09
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0a
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0b
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0c
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0d
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0e
> wikidb 2006-05-17 18:45:00: no directory, skipping
> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0f
>
> and so on...
>
>
> I'm assuming this causes rebuildImages to look through the "images" dir,
> looking for files it doesn't know about, and then add them to the database.
>
> Does this mean I need to add my files to that directory before running
> the script? If so, where? The directory apparently has a very specific
> structure, and I don't want to put images in incorrect places. Or can I
> get "rebuildImages" to look at another directory (eg.
> /wikimain/uploads), move them into the appropriate place in "images",
> and register the with the database?
>
> Once again, so many thanks for helping with this. We have (at least)
> several hundred images that need to be put into the site, and don't
> really like the thought of doing it by hand :-).

Either use the script I mentioned in a former post (which can be
stolen from SVN trunk) or set $wgHashedUploadDirectory to false and
try again.


Rob Church
_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Ken McDonald-2
Just to make the following intelligible without going through
everything, I'm trying to "batch upload" images into a local MediaWiki
site, using rebuildImages.php.  Here is Rob's reply to my most recent
question:

Either use the script I mentioned in a former post (which can be
stolen from SVN trunk) or set $wgHashedUploadDirectory to false and
try again.


Rob Church
_______________________________________________


And, unfortunately, here is what I did in "rebuildImages.php":

    function crawlMissing() {
        global $wgUploadDirectory, $wgHashedUploadDirectory;
        $wgHashedUploadDirectory = FALSE;
        if( $wgHashedUploadDirectory ) {

and the following command line execution and result:

kenneth-mcdonalds-computer:~/Sites/mediawiki-1.6.5/maintenance Ken$ php
rebuildImages.php --missing
wikidb 2006-05-17 20:06:18: crawling /Users/Ken/Sites/mediawiki-1.6.5/images
wikidb 2006-05-17 20:06:19: Surprising mime type: unknown/unknown
wikidb 2006-05-17 20:06:19: /Users/Ken/Sites/mediawiki-1.6.5/images/README
kenneth-mcdonalds-computer:~/Sites/mediawiki-1.6.5/maintenance Ken$

I get the feeling that if only I could tell rebuildImages where I
_really_ want it to get the new images from, I could get this to work.
But I don't have the PHP or MW knowledge to try that with even the
smallest likelihood of success. Can I simply set $wgUploadDirectory to
something appropriate, and if so, is the path absolute in the computer,
relative to the main MW directory, or specified somehow else :-)

Rob mentioned another possible script I could check out from SVN, and
I'll try that if I have to. But I feel this is _so_ close, surely I must
be able to get it to work...and I'm not really a development geek, so
I'd prefer to avoid messing around with Subversion.

Thanks again,
Ken

> On 17/05/06, Ken McDonald <[hidden email]> wrote:
>  
>> With many thanks to responses to previous questions, I think I _almost_
>> at the point of being able to bulk upload images. Here's the situation:
>>
>>   - My images are in a file "uploads" in the "main" wiki directory, i.e.
>> the one containing the "maintenance" directory. They all have a .jpg suffix
>>   - I have added and configured an "AdminSettings.php" file in the main
>> directory. I can now run the rebuildImages script, which I could not do
>> before.
>>
>> I get two different types of results when I try to execute the
>> rebuildImages script while in the maintenance directory:
>>
>> If I run "php rebuildImages.php", I get the result;
>>
>> wikidb 2006-05-17 18:43:07: Processing image...
>> wikidb 2006-05-17 18:43:07: Finished image... 0 of 2 rows updated
>> wikidb 2006-05-17 18:43:07: Processing oldimage...
>> wikidb 2006-05-17 18:43:07: Finished oldimage... 0 of 0 rows updated
>>
>> I have two images already in the wiki, I assume that is what is being
>> reflected in this output.
>>
>>
>> If I run "php rebuildImages.php --missing", The result is much longer,
>> starting with:
>>
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/00
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/01
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/02
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/03
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/04
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/05
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/06
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/07
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/08
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/09
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0a
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0b
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0c
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0d
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0e
>> wikidb 2006-05-17 18:45:00: no directory, skipping
>> /Users/Ken/Sites/mediawiki-1.6.5/images/0/0f
>>
>> and so on...
>>
>>
>> I'm assuming this causes rebuildImages to look through the "images" dir,
>> looking for files it doesn't know about, and then add them to the database.
>>
>> Does this mean I need to add my files to that directory before running
>> the script? If so, where? The directory apparently has a very specific
>> structure, and I don't want to put images in incorrect places. Or can I
>> get "rebuildImages" to look at another directory (eg.
>> /wikimain/uploads), move them into the appropriate place in "images",
>> and register the with the database?
>>
>> Once again, so many thanks for helping with this. We have (at least)
>> several hundred images that need to be put into the site, and don't
>> really like the thought of doing it by hand :-).
>>    
>
> Either use the script I mentioned in a former post (which can be
> stolen from SVN trunk) or set $wgHashedUploadDirectory to false and
> try again.
>
>
> Rob Church
> _______________________________________________
> MediaWiki-l mailing list
> [hidden email]
> http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
>
>
>  

_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Rob Church
On 17/05/06, Ken McDonald <[hidden email]> wrote:

> Just to make the following intelligible without going through
> everything, I'm trying to "batch upload" images into a local MediaWiki
> site, using rebuildImages.php.  Here is Rob's reply to my most recent
> question:
>
> Either use the script I mentioned in a former post (which can be
> stolen from SVN trunk) or set $wgHashedUploadDirectory to false and
> try again.
>
>
> Rob Church
> _______________________________________________
>
>
> And, unfortunately, here is what I did in "rebuildImages.php":
>
>     function crawlMissing() {
>         global $wgUploadDirectory, $wgHashedUploadDirectory;
>         $wgHashedUploadDirectory = FALSE;
>         if( $wgHashedUploadDirectory ) {
>
> and the following command line execution and result:
>
> kenneth-mcdonalds-computer:~/Sites/mediawiki-1.6.5/maintenance Ken$ php
> rebuildImages.php --missing
> wikidb 2006-05-17 20:06:18: crawling /Users/Ken/Sites/mediawiki-1.6.5/images
> wikidb 2006-05-17 20:06:19: Surprising mime type: unknown/unknown
> wikidb 2006-05-17 20:06:19: /Users/Ken/Sites/mediawiki-1.6.5/images/README
> kenneth-mcdonalds-computer:~/Sites/mediawiki-1.6.5/maintenance Ken$
>
> I get the feeling that if only I could tell rebuildImages where I
> _really_ want it to get the new images from, I could get this to work.
> But I don't have the PHP or MW knowledge to try that with even the
> smallest likelihood of success. Can I simply set $wgUploadDirectory to
> something appropriate, and if so, is the path absolute in the computer,
> relative to the main MW directory, or specified somehow else :-)
>
> Rob mentioned another possible script I could check out from SVN, and
> I'll try that if I have to. But I feel this is _so_ close, surely I must
> be able to get it to work...and I'm not really a development geek, so
> I'd prefer to avoid messing around with Subversion.

http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance

For importImages.php and importImages.inc.php, click the file name,
hit download, and save in maintenance/ with those filenames. Then run
from the command line, pointing at a dir. full of images.

I also referred to setting $wgHashedUploadDirectory to false, but
intended for this to be set in LocalSettings.

And the script sounds like it's failing because of some weird MIME
error. Psst...the bulk importer doesn't do MIME checking at the
moment.


Rob Church
_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Brion Vibber
In reply to this post by Ken McDonald-2
Ken McDonald wrote:
> And, unfortunately, here is what I did in "rebuildImages.php":
>
>     function crawlMissing() {
>         global $wgUploadDirectory, $wgHashedUploadDirectory;
>         $wgHashedUploadDirectory = FALSE;

No, that'll just break everything. :)

1) Set that in your LocalSettings.php.

2) Put all your files into the images directory.

3) run rebuildImages.php --missing

4) cross fingers

> I get the feeling that if only I could tell rebuildImages where I
> _really_ want it to get the new images from, I could get this to work.

rebuildImages is meant to update the image table for image files that are
*already* in the proper location and directory layout.

It does not move files, as it expects them to already be in place.

-- brion vibber (brion @ pobox.com)


_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l

signature.asc (257 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Ken McDonald-2
In reply to this post by Rob Church

> And the script sounds like it's failing because of some weird MIME
> error. Psst...the bulk importer doesn't do MIME checking at the
> moment.
>
>
> Rob Church
> _______________________________________________
>  
I didn't even know images could have mime types associated with them,
except via their extensions which I assume are set correctly since I can
do manual uploads :-)

In any case, Rob, I think you're right, it's time to try the batch
importers. Sigh, install SVN, figure out how to download, figure out to
configure, figure out how to use--I suppose this is the kind of thing
they pay me decent money for, but surely I'm not the only one to
question the assertion that computers make things more efficient :-)

Thanks,
Ken
_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Todd Pederzani
On 5/18/06, Ken McDonald <[hidden email]> wrote:
>
> Sigh, install SVN, figure out how to download, figure out to
> configure, figure out how to use


Why not just use your web browser along with the HTTP SVN gateway [1] that
was mentioned earlier in the thread?  There's really no need to learn SVN
when you will only be using the script once.

[1] http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/

--
Todd Pederzani
[hidden email]
_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

YES, bulk image addition works, thank you so much!

Ken McDonald-2
Rob and you are now officially my Gods. Sorry I didn't see that about
the SVN gateway before, I'm just a little overwhelmed with trying to
learn too much new stuff in too little time. In any case, just to
indicate to others how easy this is:

1) Put your images in a directory, anywhere. I called mine 'pix'.
2) Download importImages.php and importImage.inc.php from the gateway
mentioned at the end of this message, and install them in the
"maintenance" subdirectory of the wiki installation.
3) Run an appropriate command line command. For my computer, in the
"maintenance" dir, that was "php importImages.php ../pix jpg".

The first arg is the dir containing the images, second and subsequent
args are extensions indicating the file types you wish to process. These
are given _without_ the "." preceding the extension. I only used the
above command (i.e single jpg extension), so can't say anything about
other extensions.

The command will not overwrite files already in the directory; perhaps
there is a way to force such a thing? Fortunately, I don't see us
needing such a thing.

Once again, many, many thanks,
Ken

Todd Pederzani wrote:

> On 5/18/06, Ken McDonald <[hidden email]> wrote:
>  
>> Sigh, install SVN, figure out how to download, figure out to
>> configure, figure out how to use
>>    
>
>
> Why not just use your web browser along with the HTTP SVN gateway [1] that
> was mentioned earlier in the thread?  There's really no need to learn SVN
> when you will only be using the script once.
>
> [1] http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/
>
>  

_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l
Reply | Threaded
Open this post in threaded view
|

Re: Sigh, bulk image addition--so near and yet so far...

Elliott F. Cable
In reply to this post by Ken McDonald-2
Are you on a mac? It took me three steps, maybe 20 clicks and a few  
keys, to have mediawiki checked out of SVN.

On May 18, 2006, at 12:03 PM, Ken McDonald wrote:

>> And the script sounds like it's failing because of some weird MIME
>> error. Psst...the bulk importer doesn't do MIME checking at the
>> moment.
>>
>>
>> Rob Church
>> _______________________________________________
>>
> I didn't even know images could have mime types associated with them,
> except via their extensions which I assume are set correctly since  
> I can
> do manual uploads :-)
>
> In any case, Rob, I think you're right, it's time to try the batch
> importers. Sigh, install SVN, figure out how to download, figure  
> out to
> configure, figure out how to use--I suppose this is the kind of thing
> they pay me decent money for, but surely I'm not the only one to
> question the assertion that computers make things more efficient :-)
>
> Thanks,
> Ken
> _______________________________________________
> MediaWiki-l mailing list
> [hidden email]
> http://mail.wikipedia.org/mailman/listinfo/mediawiki-l

_______________________________________________
MediaWiki-l mailing list
[hidden email]
http://mail.wikipedia.org/mailman/listinfo/mediawiki-l