Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile

Yuvi Panda
Since nobody reads mobile-l


---------- Forwarded message ----------
From: Jon Robson <[hidden email]>
Date: Sat, Jun 8, 2013 at 4:17 AM
Subject: [WikimediaMobile] Number crunching: Upload errors on mobile
To: mobile-l <[hidden email]>


I ran some data crunching on a sample of 4429 photo uploads from
mobile web. In this sample 2821 uploads succeeded and 36% (1608/4429)
of attempted uploads failed. This is very high and unacceptable.

Looking closely 53% of all errors were due to problems with invalid or
'anonymous' tokens. This will occur when the client is unable to get a
token using CORS from Commons due to not being logged in there.
Luckily Chris Steipp and the rest of the platform team have pushed a
change that should significantly reduce this error:
https://gerrit.wikimedia.org/r/#/c/57662/

CentralAuth related errors (861)
####################################
Anonymous token. 598
Invalid token 263

The next big offender was 'Missing filename' accounting for 22% of all
upload errors. Unfortunately this is ambiguous as it could mean a
variety of things - it simply means that an upload was attempted and
the response didn't report the filename. I've pushed a patch to try to
understand what errors we are running into:
https://gerrit.wikimedia.org/r/67545

Other errors (467)
###################################
Missing filename 347
This file did not pass file verification 69
Blank error message given 36
The file you submitted was empty 15

There are various other errors all listed below for your enjoyment.
Some due to bad choices of name, permissions problems and attempts to
upload certain file types we do not accept. The good news is we
probably don't want these uploads to succeed as they hint at vandalism
attempts or uploads by poorly educated users.

The server problems section is worth a look though - although a small
percentage "The modification you tried to make was aborted by an
extension hook 61". These errors are occurring on the following wiki
projects:
* sv.m.wikipedia.org
* de.m.wikipedia.org
* test.m.wikipedia.org
* en.m.wikipedia.org
* ar.m.wikipedia.org
* es.m.wikipedia.org
* ja.m.wikipedia.org
* he.m.wikipedia.org
* fr.m.wikipedia.org
* nl.m.wikipedia.org
Any ideas what may be causing that error?

***
Other errors:

Users uploading with bad or unclear filenames (113)
####################################
"titleblacklist-custom-filename" 48
(https://commons.m.wikimedia.org/wiki/Template:Titleblacklist-custom-filename/en)
"titleblacklist-forbidden-edit" 37
(https://en.m.wikipedia.org/wiki/MediaWiki_talk:Titleblacklist-forbidden-edit)
Filename exists 25 (I suspect they used a common filename)
Unknown error: "titleblacklist-custom-double-apostrophe" 3

(Out of interest is there any API to check whether a filename will be accepted?)

Server problems (81):
####################################
The modification you tried to make was aborted by an extension hook 61
Database query error 10
An internal error occurred 9
error: Internal Server Error 1

Permission based errors (69)
####################################
The "autoconfirmed" right is required to edit this page 36
You have been blocked from editing 25
The "protect" right is required to edit this page 7
Unknown error: "globalblocking-ipblocked" 1

(These users should not be seeing the upload button!)

Users uploading with bad file types (17):
###########################################
Filetype not permitted: MOV 8
Filetype not permitted: webp 4
Filetype not permitted: mp3 3
Filetype not permitted: xml 1
Filetype not permitted: bmp 1

_______________________________________________
Mobile-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/mobile-l


--
Yuvi Panda T
http://yuvi.in/blog

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile

Brian Wolff
>
> The server problems section is worth a look though - although a small
> percentage "The modification you tried to make was aborted by an
> extension hook 61". These errors are occurring on the following wiki
> projects:
> * sv.m.wikipedia.org
> * de.m.wikipedia.org
> * test.m.wikipedia.org
> * en.m.wikipedia.org
> * ar.m.wikipedia.org
> * es.m.wikipedia.org
> * ja.m.wikipedia.org
> * he.m.wikipedia.org
> * fr.m.wikipedia.org
> * nl.m.wikipedia.org
> Any ideas what may be causing that error?

I suspect that is caused by UploadBlacklist extension, which
blacklists about 23 files by their sha hash. According to the config
file, there's a log at  "udp://$wmfUdp2logDest/upload-blacklist", so
you can probably check if that guess is right.

--bawolff

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile

Max Semenik
On 08.06.2013, 21:07 Brian wrote:


> I suspect that is caused by UploadBlacklist extension, which
> blacklists about 23 files by their sha hash. According to the config
> file, there's a log at  "udp://$wmfUdp2logDest/upload-blacklist", so
> you can probably check if that guess is right.

$ grep -v 'MISS' upload-blacklist.log
$

--
Best regards,
  Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [WikimediaMobile] Number crunching: Upload errors on mobile

Jon Robson
Update on error numbers:

111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token
problems.

Things are looking much better and come under 3 types of error
Here is the new breakdown

1) Errors due to bad filenames / bad filetypes

Missing filename: Bad filename 67
> This is now the biggest cause of errors. I suggest we log the file names
users are trying to upload with to identify what is going wrong. A bug is
open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544

There are various other errors which are probably not worth our time as
they suggest vandal edits:

This file did not pass file verification 4
Unknown error: "titleblacklist-forbidden-edit" 5
Unknown error: "titleblacklist-custom-filename" 1
Missing filename: Filename exists 1
Missing filename: Duplicate archive 1
Filetype not permitted: MOV 1
> The file type not permitted errors could be solved by checking the image
in preview mode correctly loaded and whether it has a width that isn't 0.

2) Errors due to tokens


Bad token name. 15
> These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and
meta.m.wikimedia.org
> 5 of them were on the uploads page
Investigating with Chris it seems this can happen when a user has logged in
on mobile, left the page for some time and thus their login has expired.
Requesting a central auth token without being logged in will cause this
error. We should explore checking login status before starting the upload
workflow and redirect the user to the login page.

Invalid token 5
> We are caching tokens that have since expired. We should explore
invalidating tokens.

3) Other errors
These bugs are all pretty mysterious and it's not clear what causes them

* The modification you tried to make was aborted by an extension hook 7
* error: 3
> Note This occurs when an error happens but the error is missing an 'info'
property.
e.g. the response is:
{ error: {} }

An internal error occurred 1


On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik <[hidden email]> wrote:

> On 08.06.2013, 21:07 Brian wrote:
>
>
>> I suspect that is caused by UploadBlacklist extension, which
>> blacklists about 23 files by their sha hash. According to the config
>> file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so
>> you can probably check if that guess is right.
>
> $ grep -v 'MISS' upload-blacklist.log
> $
>
> --
> Best regards,
> Max Semenik ([[User:MaxSem]])
>
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

--
Jon Robson
http://jonrobson.me.uk
@rakugojon
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [WikimediaMobile] Fwd: Number crunching: Upload errors on mobile

Arthur Richards-2
Nice work! Resolving the bad filename issue should get us down to a much
more acceptable error rate.


On Fri, Jun 21, 2013 at 3:19 PM, Jon Robson <[hidden email]> wrote:

> Update on error numbers:
>
> 111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token
> problems.
>
> Things are looking much better and come under 3 types of error
> Here is the new breakdown
>
> 1) Errors due to bad filenames / bad filetypes
>
> Missing filename: Bad filename 67
> > This is now the biggest cause of errors. I suggest we log the file names
> users are trying to upload with to identify what is going wrong. A bug is
> open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544
>
> There are various other errors which are probably not worth our time as
> they suggest vandal edits:
>
> This file did not pass file verification 4
> Unknown error: "titleblacklist-forbidden-edit" 5
> Unknown error: "titleblacklist-custom-filename" 1
> Missing filename: Filename exists 1
> Missing filename: Duplicate archive 1
> Filetype not permitted: MOV 1
> > The file type not permitted errors could be solved by checking the image
> in preview mode correctly loaded and whether it has a width that isn't 0.
>
> 2) Errors due to tokens
>
>
> Bad token name. 15
> > These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and
> meta.m.wikimedia.org
> > 5 of them were on the uploads page
> Investigating with Chris it seems this can happen when a user has logged
> in on mobile, left the page for some time and thus their login has expired.
> Requesting a central auth token without being logged in will cause this
> error. We should explore checking login status before starting the upload
> workflow and redirect the user to the login page.
>
> Invalid token 5
> > We are caching tokens that have since expired. We should explore
> invalidating tokens.
>
> 3) Other errors
> These bugs are all pretty mysterious and it's not clear what causes them
>
> * The modification you tried to make was aborted by an extension hook 7
> * error: 3
> > Note This occurs when an error happens but the error is missing an
> 'info' property.
> e.g. the response is:
> { error: {} }
>
> An internal error occurred 1
>
>
> On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik <[hidden email]> wrote:
> > On 08.06.2013, 21:07 Brian wrote:
> >
> >
> >> I suspect that is caused by UploadBlacklist extension, which
> >> blacklists about 23 files by their sha hash. According to the config
> >> file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so
> >> you can probably check if that guess is right.
> >
> > $ grep -v 'MISS' upload-blacklist.log
> > $
> >
> > --
> > Best regards,
> > Max Semenik ([[User:MaxSem]])
> >
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> --
> Jon Robson
> http://jonrobson.me.uk
> @rakugojon
>
> _______________________________________________
> Mobile-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>


--
Arthur Richards
Software Engineer, Mobile
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [WikimediaMobile] Fwd: Number crunching: Upload errors on mobile

Jon Robson
Update:
I did some more exploring and discovered that very long filenames can
cause the bad filename issue. I've added a margin of error to our file
length chooser algorithm:
https://gerrit.wikimedia.org/r/#/c/70223/

Whilst there I also added protection against uploading of bad file
types, filenames with double apostrophes and better logging for bad
filenames.

I suspect these changes will drastically cut the upload error rate on
mobile web.


On Mon, Jun 24, 2013 at 11:18 AM, Arthur Richards
<[hidden email]> wrote:

> Nice work! Resolving the bad filename issue should get us down to a much
> more acceptable error rate.
>
>
> On Fri, Jun 21, 2013 at 3:19 PM, Jon Robson <[hidden email]> wrote:
>>
>> Update on error numbers:
>>
>> 111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token
>> problems.
>>
>> Things are looking much better and come under 3 types of error
>> Here is the new breakdown
>>
>> 1) Errors due to bad filenames / bad filetypes
>>
>> Missing filename: Bad filename 67
>> > This is now the biggest cause of errors. I suggest we log the file names
>> > users are trying to upload with to identify what is going wrong. A bug is
>> > open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544
>>
>> There are various other errors which are probably not worth our time as
>> they suggest vandal edits:
>>
>> This file did not pass file verification 4
>> Unknown error: "titleblacklist-forbidden-edit" 5
>> Unknown error: "titleblacklist-custom-filename" 1
>> Missing filename: Filename exists 1
>> Missing filename: Duplicate archive 1
>> Filetype not permitted: MOV 1
>> > The file type not permitted errors could be solved by checking the image
>> > in preview mode correctly loaded and whether it has a width that isn't 0.
>>
>> 2) Errors due to tokens
>>
>>
>> Bad token name. 15
>> > These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and
>> > meta.m.wikimedia.org
>> > 5 of them were on the uploads page
>> Investigating with Chris it seems this can happen when a user has logged
>> in on mobile, left the page for some time and thus their login has expired.
>> Requesting a central auth token without being logged in will cause this
>> error. We should explore checking login status before starting the upload
>> workflow and redirect the user to the login page.
>>
>> Invalid token 5
>> > We are caching tokens that have since expired. We should explore
>> > invalidating tokens.
>>
>> 3) Other errors
>> These bugs are all pretty mysterious and it's not clear what causes them
>>
>> * The modification you tried to make was aborted by an extension hook 7
>> * error: 3
>> > Note This occurs when an error happens but the error is missing an
>> > 'info' property.
>> e.g. the response is:
>> { error: {} }
>>
>> An internal error occurred 1
>>
>>
>> On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik <[hidden email]> wrote:
>> > On 08.06.2013, 21:07 Brian wrote:
>> >
>> >
>> >> I suspect that is caused by UploadBlacklist extension, which
>> >> blacklists about 23 files by their sha hash. According to the config
>> >> file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so
>> >> you can probably check if that guess is right.
>> >
>> > $ grep -v 'MISS' upload-blacklist.log
>> > $
>> >
>> > --
>> > Best regards,
>> > Max Semenik ([[User:MaxSem]])
>> >
>> >
>> > _______________________________________________
>> > Wikitech-l mailing list
>> > [hidden email]
>> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>> --
>> Jon Robson
>> http://jonrobson.me.uk
>> @rakugojon
>>
>>
>> _______________________________________________
>> Mobile-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>>
>
>
>
> --
> Arthur Richards
> Software Engineer, Mobile
> [[User:Awjrichards]]
> IRC: awjr
> +1-415-839-6885 x6687



--
Jon Robson
http://jonrobson.me.uk
@rakugojon

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l