Help test chunked uploading

[Erik Moeller-4]

http://commons.wikimedia.org/wiki/Special:Preferences#mw-prefsection-uploads

There's a new user preference "chunked uploads for files over 1 MB"
for Upload Wizard.

The intent of this feature is to reliably transfer large files,
mitigating connection reliability issues. This will enable us to raise
the file size limit for all users.

Please help test this feature and report issues (performance, file
corruption, errors, etc.). The carrot is that with this preference
enabled, you can upload files <500 MB today (as opposed to <100 MB)
:-)

The feature currently only works with Upload Wizard.

More information:
http://commons.wikimedia.org/wiki/Commons:Chunked_uploads

Please report bugs on the talk page or against UploadWizard in Bugzilla:
https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions&component=UploadWizard

--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Raimond Spekking-2]

Am 04.05.2012 05:16, schrieb Erik Moeller:

> http://commons.wikimedia.org/wiki/Special:Preferences#mw-prefsection-uploads
>
> There's a new user preference "chunked uploads for files over 1 MB"
> for Upload Wizard.
>
> The intent of this feature is to reliably transfer large files,
> mitigating connection reliability issues. This will enable us to raise
> the file size limit for all users.
>
> Please help test this feature and report issues (performance, file
> corruption, errors, etc.). The carrot is that with this preference
> enabled, you can upload files <500 MB today (as opposed to <100 MB)
> :-)
>
> The feature currently only works with Upload Wizard.
>
> More information:
> http://commons.wikimedia.org/wiki/Commons:Chunked_uploads
>
> Please report bugs on the talk page or against UploadWizard in Bugzilla:
> https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions&component=UploadWizard
>

Thanks for the info, Erik.

Does anyone know, if the pywikipediabot framework supports chunked
uploading now or in the near future?

Raimond.


_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Maarten Dammers]

Hi Raimond,

Op 5-5-2012 10:46, Raimond Spekking schreef:

> Am 04.05.2012 05:16, schrieb Erik Moeller:
>> http://commons.wikimedia.org/wiki/Special:Preferences#mw-prefsection-uploads
>>
>> There's a new user preference "chunked uploads for files over 1 MB"
>> for Upload Wizard.
>>
>> The intent of this feature is to reliably transfer large files,
>> mitigating connection reliability issues. This will enable us to raise
>> the file size limit for all users.
>>
>> Please help test this feature and report issues (performance, file
>> corruption, errors, etc.). The carrot is that with this preference
>> enabled, you can upload files<500 MB today (as opposed to<100 MB)
>> :-)
>>
>> The feature currently only works with Upload Wizard.
>>
>> More information:
>> http://commons.wikimedia.org/wiki/Commons:Chunked_uploads
>>
>> Please report bugs on the talk page or against UploadWizard in Bugzilla:
>> https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions&component=UploadWizard
>>
> Thanks for the info, Erik.
>
> Does anyone know, if the pywikipediabot framework supports chunked
> uploading now or in the near future?

I was about to forward this email to the pywikipedia-l list to suggest
someone to implement this.
Based on http://www.mediawiki.org/wiki/API:Upload#Chunked_uploading this
shouldn't be too hard.

Maarten


_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Federico Leva (Nemo)]

Very nice news!
I hope we'll soon be able to reach the sweetness of the archive.org
uploader, which allowed me to upload files up to 7-8 GB smoothly and
without errors (on Chromium).

Erik Moeller, 04/05/2012 05:16:
> Please help test this feature and report issues (performance, file
> corruption, errors, etc.). The carrot is that with this preference
> enabled, you can upload files<500 MB today (as opposed to<100 MB)
> :-)

This is indeed a sweet carrot; I've tried to upload the new WMIT
WikiGuide video (190 MB) but it failed with
internal_api_error_UploadStashFileNotFoundException (took too long?).
Now I'll try again, but I had also the following (non blocking) problems
which make it unusable (or very user-unfriendly) at least on my platform:
* upload is terribly slow, averaged at 25-50 KB/s and never went above
100 (but I didn't stare at it for hours);
* especially when you add the file and start upload, it's very
resource-intensive and made my browser stall for a short while: 400 MB
of RAM used for several minutes, CPU usage hopping, a whole core waiting
for disk (?).
I suppose you're already working on these problems, but if they're
platform-specific (I used Firefox on Ubuntu) and you need more details I
can file some bugs.

Nemo

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[ProLineServer]

Hi,

On Mon, May 7, 2012 at 11:54 AM, Federico Leva (Nemo)
<[hidden email]> wrote:
> This is indeed a sweet carrot; I've tried to upload the new WMIT WikiGuide
> video (190 MB) but it failed with
> internal_api_error_UploadStashFileNotFoundException (took too long?).

I got the same error, but: The file was uploaded correctly to Commons.

> * especially when you add the file and start upload, it's very
> resource-intensive and made my browser stall for a short while: 400 MB of
> RAM used for several minutes, CPU usage hopping, a whole core waiting for
> disk (?).

Chromium on Ubuntu is just crashing, Firefox complains about a slow
script, both on Windows and Ubuntu, but it still works.

Regards, Holger

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Federico Leva (Nemo)]

Prolineserver, 07/05/2012 12:24:
>> * especially when you add the file and start upload, it's very
>> resource-intensive and made my browser stall for a short while: 400 MB of
>> RAM used for several minutes, CPU usage hopping, a whole core waiting for
>> disk (?).
>
> Chromium on Ubuntu is just crashing, Firefox complains about a slow
> script, both on Windows and Ubuntu, but it still works.

On the contrary, with Chromium 18.0.1025.142 (Build 129054 Linux) I had
a much better experience. It still failed though, filed as
https://bugzilla.wikimedia.org/show_bug.cgi?id=36587

Nemo

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Erik Moeller-4]

On Mon, May 7, 2012 at 2:54 AM, Federico Leva (Nemo) <[hidden email]> wrote:

> * especially when you add the file and start upload, it's very
> resource-intensive and made my browser stall for a short while: 400 MB of
> RAM used for several minutes, CPU usage hopping, a whole core waiting for
> disk (?).

Yeah, UW was trying to extract JPEG metadata from video files. :-P
This should be fixed once https://gerrit.wikimedia.org/r/#/c/6722/ is
reviewed and deployed.

The internal API error bug will probably be a bit harder to track
down; hazarding a guess it might be a timing issue with the copy
operation on very large files.

Can you file a bug for the upload performance issue as well? That one
is new to me.

Thanks,
Erik

--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Federico Leva (Nemo)]

Erik Moeller, 07/05/2012 19:13:

> On Mon, May 7, 2012 at 2:54 AM, Federico Leva (Nemo)  wrote:
>
>> * especially when you add the file and start upload, it's very
>> resource-intensive and made my browser stall for a short while: 400 MB of
>> RAM used for several minutes, CPU usage hopping, a whole core waiting for
>> disk (?).
>
> Yeah, UW was trying to extract JPEG metadata from video files. :-P
> This should be fixed once https://gerrit.wikimedia.org/r/#/c/6722/ is
> reviewed and deployed.
>
> The internal API error bug will probably be a bit harder to track
> down; hazarding a guess it might be a timing issue with the copy
> operation on very large files.

What about the automatic clearing of the stash? Wasn't that made more
aggressive at some point?

> Can you file a bug for the upload performance issue as well? That one
> is new to me.

Done: https://bugzilla.wikimedia.org/show_bug.cgi?id=36599

Nemo

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[とある白い猫]

Limit still is 100MB. I am willing to test but I can't with the 100MB restriction.

  -- とある白い猫  (To Aru Shiroi Neko)

On Mon, May 7, 2012 at 9:06 PM, Federico Leva (Nemo) <[hidden email]> wrote:

Erik Moeller, 07/05/2012 19:13:

On Mon, May 7, 2012 at 2:54 AM, Federico Leva (Nemo)  wrote:

* especially when you add the file and start upload, it's very
resource-intensive and made my browser stall for a short while: 400 MB of
RAM used for several minutes, CPU usage hopping, a whole core waiting for
disk (?).

Yeah, UW was trying to extract JPEG metadata from video files. :-P
This should be fixed once https://gerrit.wikimedia.org/r/#/c/6722/ is
reviewed and deployed.

The internal API error bug will probably be a bit harder to track
down; hazarding a guess it might be a timing issue with the copy
operation on very large files.

What about the automatic clearing of the stash? Wasn't that made more aggressive at some point?

Can you file a bug for the upload performance issue as well? That one
is new to me.

Done: https://bugzilla.wikimedia.org/show_bug.cgi?id=36599

Nemo

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Erik Moeller-4]

On Tue, May 8, 2012 at 1:08 PM, とある白い猫 <[hidden email]> wrote:
> Limit still is 100MB. I am willing to test but I can't with the 100MB
> restriction.

It's not -- if you get that message, it's likely that your browser
doesn't support chunked uploading (or you don't have the user
preference enabled). Which browser were you testing with?

--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[とある白い猫]

Firefox, IE and Chorme on Windows 7

• IE upload just goes on forever. It attempted to upload for over a day and nothing happened.
• Chrome just dies.
• Firefox does nothing when I pick to upload any file

I am using Special:UploadWizard to upload a 1.4gb file.

  -- とある白い猫  (To Aru Shiroi Neko)

On Tue, May 8, 2012 at 11:51 PM, Erik Moeller <[hidden email]> wrote:

On Tue, May 8, 2012 at 1:08 PM, とある白い猫 <[hidden email]> wrote:
> Limit still is 100MB. I am willing to test but I can't with the 100MB
> restriction.

It's not -- if you get that message, it's likely that your browser
doesn't support chunked uploading (or you don't have the user
preference enabled). Which browser were you testing with?

--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Federico Leva (Nemo)]

とある白い猫, 09/05/2012 23:18:
> I am using Special:UploadWizard to upload a 1.4gb file.

Erik Moeller, 04/05/2012 05:16:
 > The carrot is that with this preference
 > enabled, you can upload files<500 MB today (as opposed to<100 MB)


Nemo

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[とある白い猫]

I get the same problem with a 173 mb file.

  -- とある白い猫  (To Aru Shiroi Neko)

On Wed, May 9, 2012 at 11:56 PM, Federico Leva (Nemo) <[hidden email]> wrote:

とある白い猫, 09/05/2012 23:18:

I am using Special:UploadWizard to upload a 1.4gb file.

Erik Moeller, 04/05/2012 05:16:

> The carrot is that with this preference
> enabled, you can upload files<500 MB today (as opposed to<100 MB)

Nemo

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Erik Moeller-4]

On Fri, May 11, 2012 at 11:22 AM, とある白い猫 <[hidden email]> wrote:
> I get the same problem with a 173 mb file.

Until this change is merged and deployed:
https://gerrit.wikimedia.org/r/#/c/6722/

You'll experience browser crashes/slowness with very large files.

However, _if_ (and only if) you're able to manage to start a chunked
upload of a large file in a recent version of Firefox and Chrome,
please tell me if you're still seeing errors on the first step. We
just deployed a timeout fix that may have resolved the error message
sometimes returned on the first step.


--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[とある白い猫]

I have a no go on IE (attempts to upload forever) and Chrome (crashes the first second). I managed to upload a file on Firefox: http://commons.wikimedia.org/wiki/File:Iridescent_Glory_of_Nearby_Helix_Nebula.png

Could someone review if the upload is corrupted or not?

  -- とある白い猫  (To Aru Shiroi Neko)

On Fri, May 11, 2012 at 10:34 PM, Erik Moeller <[hidden email]> wrote:

On Fri, May 11, 2012 at 11:22 AM, とある白い猫 <[hidden email]> wrote:
> I get the same problem with a 173 mb file.

Until this change is merged and deployed:
https://gerrit.wikimedia.org/r/#/c/6722/

You'll experience browser crashes/slowness with very large files.

However, _if_ (and only if) you're able to manage to start a chunked
upload of a large file in a recent version of Firefox and Chrome,
please tell me if you're still seeing errors on the first step. We
just deployed a timeout fix that may have resolved the error message
sometimes returned on the first step.

--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l

[Erik Moeller-4]

On Fri, May 11, 2012 at 1:34 PM, Erik Moeller <[hidden email]> wrote:
> On Fri, May 11, 2012 at 11:22 AM, とある白い猫 <[hidden email]> wrote:
>> I get the same problem with a 173 mb file.
>
> Until this change is merged and deployed:
> https://gerrit.wikimedia.org/r/#/c/6722/

It is now. Please try chunked uploading again -- it should work now,
at least in recent versions of Firefox and Chrome, for files up to
500MB (some steps may take a long time to complete, but you shouldn't
get errors).


--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

_______________________________________________
Commons-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/commons-l