Thumbnails still largely broken

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Thumbnails still largely broken

Arne 'Timwi' Heizmann

Have a look at http://en.wikipedia.org/wiki/List_of_countries

More than half of the flags are broken images.

This is the same bug that I've reported millions of times before. People
keep suggesting to use action=purge to convince the server to regenerate
the image file. Well great, except doing that for all the broken flags
on this page is beyond tedious. And besides, how is a casual reader
supposed to do this.

This needs fixing quite urgently, especially seeing as it's been around
for so long.

Timwi


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Gregory Maxwell
On 9/16/07, Timwi <[hidden email]> wrote:

> Have a look at http://en.wikipedia.org/wiki/List_of_countries
> More than half of the flags are broken images.
>
> This is the same bug that I've reported millions of times before. People
> keep suggesting to use action=purge to convince the server to regenerate
> the image file. Well great, except doing that for all the broken flags
> on this page is beyond tedious. And besides, how is a casual reader
> supposed to do this.
>
> This needs fixing quite urgently, especially seeing as it's been around
> for so long.

A server ran out of disk space. It has been fixed. This is the cause
of most (all?) of the recent spike of broken thumbnails.

Without more information it is not possible to determine if there is
an additional cause of thumbnail problems which is on-going. Can you
link to your prior bug reports?

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Steve Summit
Gregory Maxwell wrote:
>On 9/16/07, Timwi <[hidden email]> wrote:
>> People keep suggesting to use action=purge to convince the server
>> to regenerate the image file. Well great, except doing that for all
>> the broken flags on this page is beyond tedious. And besides, how is
>> a casual reader supposed to do this.

Well, evidently by education: there's now a banner at the top
of every page containing a link to the purge instructions.
But is this really necessary?

> A server ran out of disk space. It has been fixed. This is the cause
> of most (all?) of the recent spike of broken thumbnails.

My understanding is that, while the disk was full, every
thumbnail that attempted to be created got created with size 0,
and is therefore now broken.  But I would assume that running a
quick and simple

        find -size 0 -print | xargs rm

(or the equivalent) on the affected server(s) would cause the
broken images to all be automatically corrected, as needed.
Am I wrong?  Or are there too many servers, all of which that
"quick and simple" command would have to be run on, or something?

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Rob Church
On 17/09/2007, Steve Summit <[hidden email]> wrote:
> (or the equivalent) on the affected server(s) would cause the
> broken images to all be automatically corrected, as needed.
> Am I wrong?  Or are there too many servers, all of which that
> "quick and simple" command would have to be run on, or something?

MediaWiki's configuration can be slightly tweaked such that all
previously cached thumbnails will be regenerated, using what we call
the "thumbnail epoch", which is like our other "cache epochs" - but
realistically, the system administrators may adopt a mixture of
solutions to try and minimise any excessive load spiking due to cache
misses or any other funky fun stuff like that.


Rob Church

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Gregory Maxwell
In reply to this post by Steve Summit
On 9/16/07, Steve Summit <[hidden email]> wrote:

> Gregory Maxwell wrote:
> >On 9/16/07, Timwi <[hidden email]> wrote:
> >> People keep suggesting to use action=purge to convince the server
> >> to regenerate the image file. Well great, except doing that for all
> >> the broken flags on this page is beyond tedious. And besides, how is
> >> a casual reader supposed to do this.
>
> Well, evidently by education: there's now a banner at the top
> of every page containing a link to the purge instructions.
> But is this really necessary?

I'm currently resisting the urge to go stabbing people.

Impacted images should now be purged from the caches. There may be a
few stragglers, but there is no reason for this to be in the site
notice.

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Nick Jenkins
In reply to this post by Rob Church
> > (or the equivalent) on the affected server(s) would cause the
> > broken images to all be automatically corrected, as needed.
> > Am I wrong?  Or are there too many servers, all of which that
> > "quick and simple" command would have to be run on, or something?
>
> MediaWiki's configuration can be slightly tweaked such that all
> previously cached thumbnails will be regenerated, using what we call
> the "thumbnail epoch", which is like our other "cache epochs" - but
> realistically, the system administrators may adopt a mixture of
> solutions to try and minimise any excessive load spiking due to cache
> misses or any other funky fun stuff like that.

Presumably though if the size of a thumbnail is 0 bytes, then MediaWiki can
safely assume that something went wrong, and that the cache is bad.
E.g. Would an "if( filesize( $thumbnail ) === 0 )" or similar test
help when determining whether to use the cache or not?

-- All the best,
Nick.

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Aryeh Gregor
On 9/16/07, Nick Jenkins <[hidden email]> wrote:
> Presumably though if the size of a thumbnail is 0 bytes, then MediaWiki can
> safely assume that something went wrong, and that the cache is bad.
> E.g. Would an "if( filesize( $thumbnail ) === 0 )" or similar test
> help when determining whether to use the cache or not?

Thumbnail caches don't involve MediaWiki at all, do they?  The change
would have to be made with Squid, I think (which, ideally, would have
gracefully refused to cache new things when it hit 0 bytes, rather
than caching zero-byte files).

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Gregory Maxwell
On 9/17/07, Simetrical <[hidden email]> wrote:

> On 9/16/07, Nick Jenkins <[hidden email]> wrote:
> > Presumably though if the size of a thumbnail is 0 bytes, then MediaWiki can
> > safely assume that something went wrong, and that the cache is bad.
> > E.g. Would an "if( filesize( $thumbnail ) === 0 )" or similar test
> > help when determining whether to use the cache or not?
>
> Thumbnail caches don't involve MediaWiki at all, do they?  The change
> would have to be made with Squid, I think (which, ideally, would have
> gracefully refused to cache new things when it hit 0 bytes, rather
> than caching zero-byte files).

This is a change that I tried to advocate a long while back because of
the intermittent issue with zero byte replies (mostly seen with large
media files .. non-thumbnail stuff). While we were using the external
popup media player I ran an additional squid just for media caching
which was configured not to cache tiny files.

The reason we can't do this everywhere is that certain responses which
we really would want to be cached are zero bytes. ... In general
anything that isn't cached is a potential outage causing agent.

For example, what if we did not cache zero byte files and I replaced a
very widely used image with one which was actually zero bytes long?

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Bryan Tong Minh
Still not working very well... I purged
<http://commons.wikimedia.org/wiki/Image:Cc-by_new_white.svg>
yesterday. It then reappeared, but had disappeared again today. And
purging it seems to make any difference to me.

On 9/17/07, Gregory Maxwell <[hidden email]> wrote:

> On 9/17/07, Simetrical <[hidden email]> wrote:
> > On 9/16/07, Nick Jenkins <[hidden email]> wrote:
> > > Presumably though if the size of a thumbnail is 0 bytes, then MediaWiki can
> > > safely assume that something went wrong, and that the cache is bad.
> > > E.g. Would an "if( filesize( $thumbnail ) === 0 )" or similar test
> > > help when determining whether to use the cache or not?
> >
> > Thumbnail caches don't involve MediaWiki at all, do they?  The change
> > would have to be made with Squid, I think (which, ideally, would have
> > gracefully refused to cache new things when it hit 0 bytes, rather
> > than caching zero-byte files).
>
> This is a change that I tried to advocate a long while back because of
> the intermittent issue with zero byte replies (mostly seen with large
> media files .. non-thumbnail stuff). While we were using the external
> popup media player I ran an additional squid just for media caching
> which was configured not to cache tiny files.
>
> The reason we can't do this everywhere is that certain responses which
> we really would want to be cached are zero bytes. ... In general
> anything that isn't cached is a potential outage causing agent.
>
> For example, what if we did not cache zero byte files and I replaced a
> very widely used image with one which was actually zero bytes long?
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> http://lists.wikimedia.org/mailman/listinfo/wikitech-l
>

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Platonides
In reply to this post by Gregory Maxwell
Gregory Maxwell wrote:

> On 9/17/07, Simetrical wrote:
>> On 9/16/07, Nick Jenkins wrote:
>>> Presumably though if the size of a thumbnail is 0 bytes, then MediaWiki can
>>> safely assume that something went wrong, and that the cache is bad.
>>> E.g. Would an "if( filesize( $thumbnail ) === 0 )" or similar test
>>> help when determining whether to use the cache or not?
>> Thumbnail caches don't involve MediaWiki at all, do they?  The change
>> would have to be made with Squid, I think (which, ideally, would have
>> gracefully refused to cache new things when it hit 0 bytes, rather
>> than caching zero-byte files).
>
> This is a change that I tried to advocate a long while back because of
> the intermittent issue with zero byte replies (mostly seen with large
> media files .. non-thumbnail stuff). While we were using the external
> popup media player I ran an additional squid just for media caching
> which was configured not to cache tiny files.

Have you a clue why it happens? Usually it would think in connections
prematurely closed, but we're giving Content-Length... (this was worked
on rfc2616, section 8.1.2.1)

> The reason we can't do this everywhere is that certain responses which
> we really would want to be cached are zero bytes. ... In general
> anything that isn't cached is a potential outage causing agent.
>
> For example, what if we did not cache zero byte files and I replaced a
> very widely used image with one which was actually zero bytes long?

You shouldn't be able to ("The file you uploaded seems to be empty...").
No problem about that. However, sometimes 0-size files appear, so maybe
we shouldn't be too happily relying on it. Can we make amane report (eg
via nagios-wm) when it sends empty files?

On the other hand, we will want to cache empty files coming from
action=raw (ie old user css & js)
Squids could detect the former because they have Cache-Control, but if
we were to patch lighttpd, it's as easy as sending them expired.


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Arne 'Timwi' Heizmann
In reply to this post by Gregory Maxwell
Gregory Maxwell wrote:
> I'm currently resisting the urge to go stabbing people.
>
> Impacted images should now be purged from the caches. There may be a
> few stragglers, but there is no reason for this to be in the site
> notice.

I do not understand your urge. If a site feature like images, which is
widely considered essential and expected to work, ceases to work, that
alone is a very good reason to have something about it in the sitenotice.


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Arne 'Timwi' Heizmann
In reply to this post by Gregory Maxwell
Gregory Maxwell wrote:
> Can you link to your prior bug reports?

I'm not sure how I'm supposed to do this given that Google is still
blocked from indexing mailing list archives for no good reason.

Timwi


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Rob Church
In reply to this post by Arne 'Timwi' Heizmann
On 17/09/2007, Timwi <[hidden email]> wrote:
> I do not understand your urge. If a site feature like images, which is
> widely considered essential and expected to work, ceases to work, that
> alone is a very good reason to have something about it in the sitenotice.

I get the impression that what Greg objects to is the (well-meaning,
but incorrect) reversion of the sitenotice back to a misleading state,
or including information about purging images which the readers should
not need to deal with. Also, the appearance of the sitenotice
indicated that someone knew about, and was dealing with, the problem,
whereas this was not the case at all.

Personally, I would disagree that images should be considered so
important on most web sites, but then, I'm one of those crazies who
believes in accessible documents, and not disadvantaging the blind,
partially-sighted and otherwise impaired.


Rob Church

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Gregory Maxwell
In reply to this post by Arne 'Timwi' Heizmann
On 9/17/07, Timwi <[hidden email]> wrote:
> Gregory Maxwell wrote:
> > Can you link to your prior bug reports?
>
> I'm not sure how I'm supposed to do this given that Google is still
> blocked from indexing mailing list archives for no good reason.

http://lists.wikimedia.org/pipermail/wikitech-l/

Use the search luke.

... Don't you have a sent-box?

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Jay Ashworth-2
In reply to this post by Rob Church
On Mon, Sep 17, 2007 at 03:26:36PM +0100, Rob Church wrote:
> Personally, I would disagree that images should be considered so
> important on most web sites, but then, I'm one of those crazies who
> believes in accessible documents, and not disadvantaging the blind,
> partially-sighted and otherwise impaired.

So instead, you'd prefer to disadvantage the (much larger) community of
users who do *not* have those impairments?  :-)

Cheers,
-- jra
--
Jay R. Ashworth                   Baylink                      [hidden email]
Designer                     The Things I Think                       RFC 2100
Ashworth & Associates     http://baylink.pitas.com                     '87 e24
St Petersburg FL USA      http://photo.imageinc.us             +1 727 647 1274

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Jay Ashworth-2
In reply to this post by Gregory Maxwell
On Mon, Sep 17, 2007 at 12:27:27AM -0400, Gregory Maxwell wrote:
> For example, what if we did not cache zero byte files and I replaced a
> very widely used image with one which was actually zero bytes long?

Well, I believe his assertion is "that's not a valid thing to ever want
to do".  Could you synthesize a situation in which it would be?

Cheers,
-- jra
--
Jay R. Ashworth                   Baylink                      [hidden email]
Designer                     The Things I Think                       RFC 2100
Ashworth & Associates     http://baylink.pitas.com                     '87 e24
St Petersburg FL USA      http://photo.imageinc.us             +1 727 647 1274

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Thomas Dalton
In reply to this post by Gregory Maxwell
On 17/09/2007, Gregory Maxwell <[hidden email]> wrote:

> On 9/17/07, Timwi <[hidden email]> wrote:
> > Gregory Maxwell wrote:
> > > Can you link to your prior bug reports?
> >
> > I'm not sure how I'm supposed to do this given that Google is still
> > blocked from indexing mailing list archives for no good reason.
>
> http://lists.wikimedia.org/pipermail/wikitech-l/
>
> Use the search luke.
>
> ... Don't you have a sent-box?

Shouldn't prior bug reports be on Bugzilla rather than the mailing list, anyway?

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Arne 'Timwi' Heizmann
In reply to this post by Rob Church
Rob Church wrote:
> On 17/09/2007, Timwi <[hidden email]> wrote:
>> I do not understand your urge. If a site feature like images, which is
>> widely considered essential and expected to work, ceases to work, that
>> alone is a very good reason to have something about it in the sitenotice.
>
> I get the impression that what Greg objects to is the (well-meaning,
> but incorrect) reversion of the sitenotice back to a misleading state,
> or including information about purging images which the readers should
> not need to deal with.

You are right that readers should not need to deal with purging images.
That is why images need to be kept working normally. Given that they
were not working normally, and purging was the only means to bring up
some of them, it makes sense to help clueless readers in this direction
for the duration of the failure.

> Personally, I would disagree that images should be considered so
> important on most web sites, but then, I'm one of those crazies who
> believes in accessible documents, and not disadvantaging the blind,
> partially-sighted and otherwise impaired.

This is the kind of attitude that could make me want to stab people.
Just because blind people exist, we are all supposed to pretend to be
blind. It is somehow no longer legitimate to want to bring up a list of
country flags in order to find one I saw elsewhere, just because a blind
person wouldn't be able to do that because they're blind, which I'm not.

Timwi


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Arne 'Timwi' Heizmann
In reply to this post by Gregory Maxwell
Gregory Maxwell wrote:
> On 9/17/07, Timwi <[hidden email]> wrote:
>> Gregory Maxwell wrote:
>>> Can you link to your prior bug reports?
>> I'm not sure how I'm supposed to do this given that Google is still
>> blocked from indexing mailing list archives for no good reason.
>
> http://lists.wikimedia.org/pipermail/wikitech-l/
>
> Use the search luke.

Such a crap search thing. If I type "Timwi" as one of the search terms,
surely it should be prioritising postings by myself. It seems to be
searching only the bodies. It's simply no match for Google.

Here are two relevant results I found:

1) A reply that advocates the "action=purge" non-solution:
http://lists.wikimedia.org/mailman/htdig/wikitech-l/2007-June/031635.html

2) A posting that acknowledges that the problem exists (but no-one knows
anything about it and therefore cannot fix it):
http://lists.wikimedia.org/mailman/htdig/wikitech-l/2007-July/032471.html

If you want better results, please re-enable Google indexing.

Timwi


_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: Thumbnails still largely broken

Gregory Maxwell
On 9/17/07, Timwi <[hidden email]> wrote:

> Here are two relevant results I found:
> 1) A reply that advocates the "action=purge" non-solution:
> http://lists.wikimedia.org/mailman/htdig/wikitech-l/2007-June/031635.html
>
> 2) A posting that acknowledges that the problem exists (but no-one knows
> anything about it and therefore cannot fix it):
> http://lists.wikimedia.org/mailman/htdig/wikitech-l/2007-July/032471.html

These issues are completely unrelated to the problem we expirenced this weekend.

_______________________________________________
Wikitech-l mailing list
[hidden email]
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
12