Bot upload of files over 100MB

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Bot upload of files over 100MB

Daniel Mietchen
Hi,

my bot[1] occasionally stumbles upon files that are above 100MB and
thus does not upload them[2]. What do I have to do to get it set up
for handling these files too?

Thanks for any pointers,

Daniel

[1] http://commons.wikimedia.org/wiki/User:Open_Access_Media_Importer_Bot
[2] https://github.com/erlehmann/open-access-media-importer/issues/22#issuecomment-20009173

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Techman224-3
Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.

Techmn224

On 2013-06-25, at 4:43 PM, Daniel Mietchen <[hidden email]> wrote:

> Hi,
>
> my bot[1] occasionally stumbles upon files that are above 100MB and
> thus does not upload them[2]. What do I have to do to get it set up
> for handling these files too?
>
> Thanks for any pointers,
>
> Daniel
>
> [1] http://commons.wikimedia.org/wiki/User:Open_Access_Media_Importer_Bot
> [2] https://github.com/erlehmann/open-access-media-importer/issues/22#issuecomment-20009173
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Jeremy Baron
In reply to this post by Daniel Mietchen
On Jun 25, 2013 5:44 PM, "Daniel Mietchen" <[hidden email]>
wrote:
> my bot[1] occasionally stumbles upon files that are above 100MB and
> thus does not upload them[2]. What do I have to do to get it set up
> for handling these files too?

this looks like the relevant section:

https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading

I don't know the current settings; you might need to enable chunked uploads
in the MediaWiki prefs for the user you're uploading as.

-Jeremy
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
OQ
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

OQ
I would assume given the previous reply that regardless of upload method,
chunked or otherwise, there is still the hard limit of how big the
resultant file can be.


On Tue, Jun 25, 2013 at 5:49 PM, Jeremy Baron <[hidden email]> wrote:

> On Jun 25, 2013 5:44 PM, "Daniel Mietchen" <[hidden email]
> >
> wrote:
> > my bot[1] occasionally stumbles upon files that are above 100MB and
> > thus does not upload them[2]. What do I have to do to get it set up
> > for handling these files too?
>
> this looks like the relevant section:
>
> https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading
>
> I don't know the current settings; you might need to enable chunked uploads
> in the MediaWiki prefs for the user you're uploading as.
>
> -Jeremy
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Tilman Bayer
With the https://commons.wikimedia.org/wiki/Commons:Chunked_uploads
enabled, users can currently upload files up to 500MB, when using a
supported browser (e.g. a current Firefox or Chrome).

I think Daniel's question was about how to make this method work for a
bot too, i.e. via the API rather than in a browser.

On Tue, Jun 25, 2013 at 2:52 PM, OQ <[hidden email]> wrote:

> I would assume given the previous reply that regardless of upload method,
> chunked or otherwise, there is still the hard limit of how big the
> resultant file can be.
>
>
> On Tue, Jun 25, 2013 at 5:49 PM, Jeremy Baron <[hidden email]> wrote:
>
>> On Jun 25, 2013 5:44 PM, "Daniel Mietchen" <[hidden email]
>> >
>> wrote:
>> > my bot[1] occasionally stumbles upon files that are above 100MB and
>> > thus does not upload them[2]. What do I have to do to get it set up
>> > for handling these files too?
>>
>> this looks like the relevant section:
>>
>> https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading
>>
>> I don't know the current settings; you might need to enable chunked uploads
>> in the MediaWiki prefs for the user you're uploading as.
>>
>> -Jeremy
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Tilman Bayer
Senior Operations Analyst (Movement Communications)
Wikimedia Foundation
IRC (Freenode): HaeB

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Brian Wolff
When browsing through commons, I happened to stumble upon
https://commons.wikimedia.org/wiki/User:Smallman12q/PyCJWiki which
appears to be a python bot that uses chunked uploading, and thus work
with files up to 500 mb big. Perhaps it would be helpful to you.

--bawolff

On 6/25/13, Tilman Bayer <[hidden email]> wrote:

> With the https://commons.wikimedia.org/wiki/Commons:Chunked_uploads
> enabled, users can currently upload files up to 500MB, when using a
> supported browser (e.g. a current Firefox or Chrome).
>
> I think Daniel's question was about how to make this method work for a
> bot too, i.e. via the API rather than in a browser.
>
> On Tue, Jun 25, 2013 at 2:52 PM, OQ <[hidden email]> wrote:
>> I would assume given the previous reply that regardless of upload method,
>> chunked or otherwise, there is still the hard limit of how big the
>> resultant file can be.
>>
>>
>> On Tue, Jun 25, 2013 at 5:49 PM, Jeremy Baron <[hidden email]>
>> wrote:
>>
>>> On Jun 25, 2013 5:44 PM, "Daniel Mietchen"
>>> <[hidden email]
>>> >
>>> wrote:
>>> > my bot[1] occasionally stumbles upon files that are above 100MB and
>>> > thus does not upload them[2]. What do I have to do to get it set up
>>> > for handling these files too?
>>>
>>> this looks like the relevant section:
>>>
>>> https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading
>>>
>>> I don't know the current settings; you might need to enable chunked
>>> uploads
>>> in the MediaWiki prefs for the user you're uploading as.
>>>
>>> -Jeremy
>>> _______________________________________________
>>> Wikitech-l mailing list
>>> [hidden email]
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Tilman Bayer
> Senior Operations Analyst (Movement Communications)
> Wikimedia Foundation
> IRC (Freenode): HaeB
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Greg Grossmeier-2
In reply to this post by Techman224-3
<quote name="Techman224" date="2013-06-25" time="16:47:39 -0500">
> Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.

I thought it was set to 500mb?
https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edce951a2eb14ae6385aa1eac24d0/wmf-config%2FInitialiseSettings.php#L10185

Greg

--
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Techman224-3
https://commons.wikimedia.org/w/index.php?title=Special:Upload shows 100 MB.

I turned on chunked uploads and it still shows 100 MB.

Techman224

On 2013-06-25, at 5:13 PM, Greg Grossmeier <[hidden email]> wrote:

> <quote name="Techman224" date="2013-06-25" time="16:47:39 -0500">
>> Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.
>
> I thought it was set to 500mb?
> https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edce951a2eb14ae6385aa1eac24d0/wmf-config%2FInitialiseSettings.php#L10185
>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Techman224-3
There's a comment for that variable, "Only affects URL uploads; web uploads are enforced by PHP."

Techman224

On 2013-06-25, at 5:18 PM, Techman224 <[hidden email]> wrote:

> https://commons.wikimedia.org/w/index.php?title=Special:Upload shows 100 MB.
>
> I turned on chunked uploads and it still shows 100 MB.
>
> Techman224
>
> On 2013-06-25, at 5:13 PM, Greg Grossmeier <[hidden email]> wrote:
>
>> <quote name="Techman224" date="2013-06-25" time="16:47:39 -0500">
>>> Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.
>>
>> I thought it was set to 500mb?
>> https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edce951a2eb14ae6385aa1eac24d0/wmf-config%2FInitialiseSettings.php#L10185
>>
>> Greg
>>
>> --
>> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
>> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Brian Wolff
In reply to this post by Greg Grossmeier-2
On 6/25/13, Greg Grossmeier <[hidden email]> wrote:

> <quote name="Techman224" date="2013-06-25" time="16:47:39 -0500">
>> Right now the wiki is setup to only allow up to 100 MB files. The only way
>> I see to upload a file greater than that is to file a bugzilla request and
>> a system administrator with shell access can manually upload it using a
>> script.
>
> I thought it was set to 500mb?
> https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edce951a2eb14ae6385aa1eac24d0/wmf-config%2FInitialiseSettings.php#L10185
>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Well as the comment says:
"// Only affects URL uploads; web uploads are enforced by PHP."

That particular variable limits chunked uploads, and upload by url
(which is only enabled for flickr). Direct normal (non-chunked)
uploads get limited by the smallest of: MW config variable,
upload_max_filesize, and post_max_size and hence are limited to 100mb.

Note: Special:Upload is always non-chunked. Upload Wizard uses chunked
based on a preference, and you can use either method if you do things
yourself using the API.

--bawolff

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Techman224-3
In reply to this post by Techman224-3
Oh yea, I forgot it was for the Upload Wizard. It doesn't say the maximum file size allowed.

Techman224

On 2013-06-25, at 5:22 PM, Techman224 <[hidden email]> wrote:

> There's a comment for that variable, "Only affects URL uploads; web uploads are enforced by PHP."
>
> Techman224
>
> On 2013-06-25, at 5:18 PM, Techman224 <[hidden email]> wrote:
>
>> https://commons.wikimedia.org/w/index.php?title=Special:Upload shows 100 MB.
>>
>> I turned on chunked uploads and it still shows 100 MB.
>>
>> Techman224
>>
>> On 2013-06-25, at 5:13 PM, Greg Grossmeier <[hidden email]> wrote:
>>
>>> <quote name="Techman224" date="2013-06-25" time="16:47:39 -0500">
>>>> Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.
>>>
>>> I thought it was set to 500mb?
>>> https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edce951a2eb14ae6385aa1eac24d0/wmf-config%2FInitialiseSettings.php#L10185
>>>
>>> Greg
>>>
>>> --
>>> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
>>> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list
>>> [hidden email]
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Daniel Mietchen
In reply to this post by Brian Wolff
Thanks to all the replies so far. Looks like chunked upload is the way to go.

Cheers,

Daniel


On Wed, Jun 26, 2013 at 12:06 AM, Brian Wolff <[hidden email]> wrote:

> When browsing through commons, I happened to stumble upon
> https://commons.wikimedia.org/wiki/User:Smallman12q/PyCJWiki which
> appears to be a python bot that uses chunked uploading, and thus work
> with files up to 500 mb big. Perhaps it would be helpful to you.
>
> --bawolff
>
> On 6/25/13, Tilman Bayer <[hidden email]> wrote:
>> With the https://commons.wikimedia.org/wiki/Commons:Chunked_uploads
>> enabled, users can currently upload files up to 500MB, when using a
>> supported browser (e.g. a current Firefox or Chrome).
>>
>> I think Daniel's question was about how to make this method work for a
>> bot too, i.e. via the API rather than in a browser.
>>
>> On Tue, Jun 25, 2013 at 2:52 PM, OQ <[hidden email]> wrote:
>>> I would assume given the previous reply that regardless of upload method,
>>> chunked or otherwise, there is still the hard limit of how big the
>>> resultant file can be.
>>>
>>>
>>> On Tue, Jun 25, 2013 at 5:49 PM, Jeremy Baron <[hidden email]>
>>> wrote:
>>>
>>>> On Jun 25, 2013 5:44 PM, "Daniel Mietchen"
>>>> <[hidden email]
>>>> >
>>>> wrote:
>>>> > my bot[1] occasionally stumbles upon files that are above 100MB and
>>>> > thus does not upload them[2]. What do I have to do to get it set up
>>>> > for handling these files too?
>>>>
>>>> this looks like the relevant section:
>>>>
>>>> https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading
>>>>
>>>> I don't know the current settings; you might need to enable chunked
>>>> uploads
>>>> in the MediaWiki prefs for the user you're uploading as.
>>>>
>>>> -Jeremy
>>>> _______________________________________________
>>>> Wikitech-l mailing list
>>>> [hidden email]
>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>> _______________________________________________
>>> Wikitech-l mailing list
>>> [hidden email]
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
>>
>> --
>> Tilman Bayer
>> Senior Operations Analyst (Movement Communications)
>> Wikimedia Foundation
>> IRC (Freenode): HaeB
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Bot upload of files over 100MB

Federico Leva (Nemo)
We have several tools and bots using chunked upload via the API for
files over 100 MB. Daniel, please report back if you encounter
https://bugzilla.wikimedia.org/show_bug.cgi?id=36587 ; it seemed fixed
at some point but Fastily (developer of one of those tools) reopened it.

Nemo

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l