scaled media (thumbs) as *temporary* files, not stored forever

classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Platonides
Ariel wrote:
> But there's plenty of "odd" sizes with lots of thumbs too. For example,
> over 65K files with width 181px

181px images are tipically made when 180px images are "bad", with the
squid/scaler refusing to deliver the latest version. Adding 1px there
forces a regeneration of the file, solving the issue.
Going up to 65K seems too much, though.


On 31/08/12 22:35, Roan Kattouw wrote:

> On Fri, Aug 31, 2012 at 1:22 PM, Jeremy Baron <[hidden email]> wrote:
>> I believe the status quo is no thumbs (of any size) are generated at
>> upload time. They are all just done on demand.
>>
> That's correct in theory. In practice (for uploads using
> Special:Upload at least) the user is redirected to the file
> description page after a successful upload, which contains a thumb of
> the file of a given size (800px?), which is then immediately generated
> on demand :) . Special:UploadWizard has similar behavior: the success
> page contains 200px(?) thumbs of all the images the user uploaded.
>
> Roan

Plus another tiny one at the file history section.

I remember seeing a change about rounding the requested image size some
months ago, so the "request any size" may not be true any longer.


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Brion Vibber
In reply to this post by Bartosz Dziewoński
On Fri, Aug 31, 2012 at 6:17 PM, Bartosz Dziewoński <[hidden email]>wrote:

> Please don't. The current syntax is nice, concise, consistent and not
> overflowing with special characters. The proposed one is verbose and
> "looks technical".


The current syntax is actually hard to machine-parse, with lots of
language-specific overrides and weird options that combine in non-obvious
ways. Not to mention that it overrides the simple link syntax... I wish
when we'd renamed Image: to File: that we'd left the magic behavior only on
the Image: alias so that links would be rationalized. :(

Keep in mind also that in the glorious unicorn-filled future most people
who aren't doing low-level template work will rarely see the markup.
They'll just push a button to insert an image.


> But defaulting to thumb seems like a good idea to
> me (but we ought to make some usage stats first :) ).
>

Yeah... changing existing behavior gets scary. :)

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

George William Herbert
On Fri, Aug 31, 2012 at 4:38 PM, Brion Vibber <[hidden email]> wrote:
> Keep in mind also that in the glorious unicorn-filled future most people
> who aren't doing low-level template work will rarely see the markup.
> They'll just push a button to insert an image.

Is that really a rainbow coming out of the Unicorn, Brion?  It seems
like an unlikely place to...... Nevermind.

>> But defaulting to thumb seems like a good idea to
>> me (but we ought to make some usage stats first :) ).
>>
>
> Yeah... changing existing behavior gets scary. :)

Usage stats win.  So - how can we get those?


--
-george william herbert
[hidden email]

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

David Gerard-2
In reply to this post by Brion Vibber
On 1 September 2012 00:38, Brion Vibber <[hidden email]> wrote:

> The current syntax is actually hard to machine-parse, with lots of
> language-specific overrides and weird options that combine in non-obvious
> ways. Not to mention that it overrides the simple link syntax... I wish
> when we'd renamed Image: to File: that we'd left the magic behavior only on
> the Image: alias so that links would be rationalized. :(


PROBLEM: Thumbnails are taking up a lot of disk space.
SOLUTION: Completely revamp image syntax in a backwards-incompatible manner.

There's something excessive there ...

Is it really the case that the server can't get atime of the images?
iIf it can, then we *know* which thumbs may as well be trashed and
regenerated as and when someone cares. If it can't, why not?


- d.

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Brion Vibber
On Fri, Aug 31, 2012 at 9:31 PM, David Gerard <[hidden email]> wrote:

> On 1 September 2012 00:38, Brion Vibber <[hidden email]> wrote:
>
> > The current syntax is actually hard to machine-parse, with lots of
> > language-specific overrides and weird options that combine in non-obvious
> > ways. Not to mention that it overrides the simple link syntax... I wish
> > when we'd renamed Image: to File: that we'd left the magic behavior only
> on
> > the Image: alias so that links would be rationalized. :(
>
>
> PROBLEM: Thumbnails are taking up a lot of disk space.
> SOLUTION: Completely revamp image syntax in a backwards-incompatible
> manner.
>
> There's something excessive there ...
>

Nah, we're just years behind on modernizing image handling. Such
improvements need to be done at some point regardless of this particular
question, but it would help with it in certain ways.

Is it really the case that the server can't get atime of the images?
> iIf it can, then we *know* which thumbs may as well be trashed and
> regenerated as and when someone cares. If it can't, why not?
>

I'd tend to think that image scaler CPU time is more precious than disk
space used by thumbs; in theory we don't actually need to "store" thumbs if
we just cache them and have a suitably large cache. A caching HTTP proxy
should have some sort of LRU-or-other system to discard old things that
aren't being used; my main worry would be about whether the system can
survive the load of a cache being cleared (say due to downtime, upgrades,
incompatible storage formats for upgrades, or whatever).

Of course if you don't have to scale anything on demand on the server side,
suddenly the entire problem disappears. Just something to think about.

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Robert Rohde
In reply to this post by Jon Robson
On Fri, Aug 31, 2012 at 10:21 AM, Jon Robson <[hidden email]> wrote:

>> >
>> > 1.  We could generate and keep only certain sizes, tossing the rest.
>> >
>>
>> Heck yes. Generate some standard sizes at upload time and let the browser
>> scale if a funny size is demanded. Modern browsers scale photos nicely,
> not
>> like the nearest-neighbor ugliness from 2002.
>
> +1 I was very surprised to learn any thumbnail sizes could be generated. We
> should standardise on a tiny, small,medium high and original resolutions. 5
> sizes seems more than enough.

Don't forget that Mediawiki is also more than just Wikipedia.  I've
worked with Mediawiki sites that were more graphics-focused and more
detailed on there layout demands than Wikipedia, and seen far more
than five sizes in play.  The diversity of sizes can be surprising,
but isn't necessarily illogical, for example, one might specify sizes
so that 2 or 3 or 4 images neatly span a 670px main column on a site
formatted for 800px displays.

Such things happen in the larger universe of Mediawiki sites, even
though they might be frowned upon on Wikipedia, etc. Personally, I've
found Mediawiki's flexibility to generate any size on demand to be
very useful.  And I would agree with Isarra that even though browsers
are much better than they were at scaling, it still results in a loss
of detail if both the server and browser are asked to do scaling.

So, my strong preference would be for solutions that don't assume a
few sizes are good enough for everyone.  At the very least, Mediawiki
should be configurable to preserve the current behavior as an option
(i.e. allowing any requested size to be produced by the server).

I would also note that the standard image description page currently
provides direct links to images scaled to "Other resolutions".  This
means that typical large image will come in at least 5 sizes (320px,
640px, 800px, 1024px, 1280px) plus the typical "thumb" size (220px on
most Wikipedias, I believe), as well as the full-size resolution.

That said, I agree that finding a way to expire old thumbs, or rarely
accessed thumbs, is definitely a good idea.

-Robert Rohde

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

MZMcBride-2
In reply to this post by George William Herbert
George Herbert wrote:
>>> But defaulting to thumb seems like a good idea to
>>> me (but we ought to make some usage stats first :) ).
>>
>> Yeah... changing existing behavior gets scary. :)
>
> Usage stats win.  So - how can we get those?

How much consideration is (or should be) given to people who hotlink images
from Wikimedia Commons or another Wikimedia wiki?

MZMcBride



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Tim Starling-2
In reply to this post by Ariel Glenn WMF
On 31/08/12 22:36, Ariel T. Glenn wrote:
> So there are some things we could change:
>
> 1.  We could generate and keep only certain sizes, tossing the rest.
> 2.  We could keep *nothing*, scaling all media as required.
> 3.  We could have a cron job that was clever about tossing thumbs every
> day (not sure how easy it would be to be clever).
> 4.  ??

I'll go for option 4. You can't delete the images from the backend
while they are still in Squid, because then they would not be purged
when the image is updated or action=purge is requested. In fact, that
is one of only two reasons for the existence of the backend thumbnail
store on Wikimedia. The thumbnail backend could be replaced by a text
file that stores a list of thumbnail filenames which were sent to
Squid within a window equivalent to the expiry time sent in the
Cache-Control header.

The other reason for the existence of the backend thumbnail store is
to transport images from the thumbnail scalers to the 404 handler. For
that purpose, the image only needs to exist in the backend for a few
seconds. It could be replaced by a better 404 handler, that sends
thumbnails directly by HTTP. Maybe the Swift one does that already.

-- Tim Starling



_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Thomas Dalton
In reply to this post by Brion Vibber
On Aug 31, 2012 11:52 PM, "Brion Vibber" <[hidden email]> wrote:
>
> * Definitely don't have "left" "right" or "center" options.

Can you elaborate on that? The positioning of images can make a big
difference to how a page looks. Do you really think you can automate it in
a way that makes pages always look good? It's also useful to be able to
know where an image is going to be displayed so you can say thing like "as
can be seen in the image to the right".

Getting images to work well on phones and tablets probably requires more
user control, not less. It would be useful to be able to specify whether an
image is vital to the article and should always be displayed or if it is
just there to look nice and can be skipped if there isn't much screen
space. (Sensible defaults are a must, of course.)
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Tim Starling-2
In reply to this post by Derric Atzrott
On 01/09/12 03:48, Derric Atzrott wrote:
> I wouldn't rely too much on the Squids.  They do an excellent job, but thinking
> of those of us who use Mediawiki outside the foundation, I would much rather see
> an intelligent thumbnail generating and caching scheme that doesn't rely on
> Squid being present.

We've been relying on Squid being present for years. Squid does a good
job of LRU caching, it's fast and the size is ample. It's persistent,
storing thumbnails on disk (some servers have flash). It would take a
lot of work to do something as good as it in the backend.

If we wanted a greater cold cache capacity, then we could add more
image scalers.

Video transcoding is an exception, but it needs to be handled
separately anyway, because you can't run an hour-long transcoding job
with no concurrency control, no error handling and no user feedback
(progress bars etc.).

-- Tim Starling


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Brion Vibber
In reply to this post by Thomas Dalton
On Sun, Sep 2, 2012 at 8:25 PM, Thomas Dalton <[hidden email]>wrote:

> On Aug 31, 2012 11:52 PM, "Brion Vibber" <[hidden email]> wrote:
> >
> > * Definitely don't have "left" "right" or "center" options.
>
> Can you elaborate on that? The positioning of images can make a big
> difference to how a page looks. Do you really think you can automate it in
> a way that makes pages always look good?


Looking at say https://en.wikipedia.org/wiki/San_Francisco the positioning
of most photos to left or right floats seems fairly random; where both left
and right are used it seems to be a manual hack to keep images from
stacking on top of other images or tables, based on typical screen sizes.

Would an automatic gallery layout look as good and be as usable? Honestly,
it might; I don't see much that's meaningful about the way these images are
laid out that would be lost by a different layout.

Would it look the same exactly? No, but who cares?

Should it lay out in right and left alignment? Maybe not -- maybe it should
use horizontal space and avoid floats? Maybe it should use a dedicated
right-side gutter (left on RTL)?

Maybe we should at least think about it.

It's also useful to be able to
> know where an image is going to be displayed so you can say thing like "as
> can be seen in the image to the right".
>

No space to left or right on mobile; safer not to rely on such positioning
being consistently relatable.

Consider also a hyperlink instead of a vague direction when referencing
something. :)

Getting images to work well on phones and tablets probably requires more
> user control, not less. It would be useful to be able to specify whether an
> image is vital to the article and should always be displayed or if it is
> just there to look nice and can be skipped if there isn't much screen
> space. (Sensible defaults are a must, of course.)
>

Indeed, distinguishing between different types of things can help -- and I
think would help far more than any manual positioning in the majority of
cases that aren't icons or otherwise explicitly inline in text or a table.

Note that tables, infoboxes, etc have the same issues with positioning,
floating, referencing, and whatnot. And like panoramic images, they
sometimes don't fit on small screens well; that's another thing to think
about.

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Tei-2
In reply to this post by Isarra Yos
On 31 August 2012 19:45, Isarra Yos <[hidden email]> wrote:

> On 31/08/2012 08:57, Brion Vibber wrote:
>>
>> Heck yes. Generate some standard sizes at upload time and let the browser
>> scale if a funny size is demanded. Modern browsers scale photos nicely, not
>> like the nearest-neighbor ugliness from 2002.
>
>
> As a graphist, I must say this does not seem like a good idea. Only
> rendering certain sizes and having the browser then scale the weird ones
> will still result in fuzzy images, because no matter how good the renderer,
> every time a bitmap image is scaled down, sharpness is lost. This is part of
> why there is so much emphasis placed on using vectors even in a static
> environment - with those, the first scale down is also avoided, and there is
> a very visible difference in clarity even there. But while only rendering
> certain sizes and then having the browser scale those would defeat that
> purpose, having to scale down bitmaps twice would look even worse,
> regardless of subject.
>
> --
> -— Isarra

A possible scenario where a human intervention is always needed is
generating icons from svg files that draw flags or ..icons.

A 16x16 pixels USA flag rendered from a SVG by some naive rescaling
(mipmaping?) will look worse than wrong. Perhaps you still want to
have this icon generated from or inspired by the SVG file.
In videogames this sort of problems are sometimes solved by having all
the scaled versions precalculated in a single file: mipmaps.

http://en.wikipedia.org/wiki/Mipmap

Modern videogames uses other more advanced techniques, but the beauty
of mipmaps is that can be artist edited (perhaps the artist can edit
the 16x16 pixels version to still make sense.



--
--
ℱin del ℳensaje.

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Magnus Manske-2
In reply to this post by Brion Vibber
On Mon, Sep 3, 2012 at 5:45 AM, Brion Vibber <[hidden email]> wrote:

> On Sun, Sep 2, 2012 at 8:25 PM, Thomas Dalton <[hidden email]>wrote:
>
>> On Aug 31, 2012 11:52 PM, "Brion Vibber" <[hidden email]> wrote:
>> >
>> > * Definitely don't have "left" "right" or "center" options.
>>
>> Can you elaborate on that? The positioning of images can make a big
>> difference to how a page looks. Do you really think you can automate it in
>> a way that makes pages always look good?
>
>
> Looking at say https://en.wikipedia.org/wiki/San_Francisco the positioning
> of most photos to left or right floats seems fairly random; where both left
> and right are used it seems to be a manual hack to keep images from
> stacking on top of other images or tables, based on typical screen sizes.
>
> Would an automatic gallery layout look as good and be as usable? Honestly,
> it might; I don't see much that's meaningful about the way these images are
> laid out that would be lost by a different layout.
>
> Would it look the same exactly? No, but who cares?
>
> Should it lay out in right and left alignment? Maybe not -- maybe it should
> use horizontal space and avoid floats? Maybe it should use a dedicated
> right-side gutter (left on RTL)?
>
> Maybe we should at least think about it.
>
> It's also useful to be able to
>> know where an image is going to be displayed so you can say thing like "as
>> can be seen in the image to the right".
>>
>
> No space to left or right on mobile; safer not to rely on such positioning
> being consistently relatable.
>
> Consider also a hyperlink instead of a vague direction when referencing
> something. :)
>
> Getting images to work well on phones and tablets probably requires more
>> user control, not less. It would be useful to be able to specify whether an
>> image is vital to the article and should always be displayed or if it is
>> just there to look nice and can be skipped if there isn't much screen
>> space. (Sensible defaults are a must, of course.)
>>
>
> Indeed, distinguishing between different types of things can help -- and I
> think would help far more than any manual positioning in the majority of
> cases that aren't icons or otherwise explicitly inline in text or a table.
>
> Note that tables, infoboxes, etc have the same issues with positioning,
> floating, referencing, and whatnot. And like panoramic images, they
> sometimes don't fit on small screens well; that's another thing to think
> about.
>
> -- brion
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Illustrating the problem of manual right/left aligned thumbnails and
elements by using slightly different CSS:

http://toolserver.org/~magnus/redefined/?page=San%20Francisco

Magnus

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Isarra Yos
On 03/09/2012 05:33, Magnus Manske wrote:
> Illustrating the problem of manual right/left aligned thumbnails and
> elements by using slightly different CSS:
>
> http://toolserver.org/~magnus/redefined/?page=San%20Francisco
>
> Magnus

That looks like as much a problem with the general design there as with
the use of images on the page itself, though.

--
-— Isarra


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Thomas Dalton
On 3 September 2012 17:10, Isarra Yos <[hidden email]> wrote:

> On 03/09/2012 05:33, Magnus Manske wrote:
>>
>> Illustrating the problem of manual right/left aligned thumbnails and
>> elements by using slightly different CSS:
>>
>> http://toolserver.org/~magnus/redefined/?page=San%20Francisco
>>
>> Magnus
>
> That looks like as much a problem with the general design there as with the
> use of images on the page itself, though.

I agree. The skin not wrapping text around the images isn't caused by
users having a choice of what side of the page to put the image on.
It's caused by the skin being badly designed...

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Platonides
In reply to this post by Tim Starling-2
On 03/09/12 02:59, Tim Starling wrote:

> I'll go for option 4. You can't delete the images from the backend
> while they are still in Squid, because then they would not be purged
> when the image is updated or action=purge is requested. In fact, that
> is one of only two reasons for the existence of the backend thumbnail
> store on Wikimedia. The thumbnail backend could be replaced by a text
> file that stores a list of thumbnail filenames which were sent to
> Squid within a window equivalent to the expiry time sent in the
> Cache-Control header.
>
> The other reason for the existence of the backend thumbnail store is
> to transport images from the thumbnail scalers to the 404 handler. For
> that purpose, the image only needs to exist in the backend for a few
> seconds. It could be replaced by a better 404 handler, that sends
> thumbnails directly by HTTP. Maybe the Swift one does that already.
>
> -- Tim Starling

The second one seems easy to fix. The first one should IMHO be fixed in
squid/varnish by allowing wildcard purges (ie. PURGE
/wikipedia/commons/thumb/5/5c/Tim_starling.jpg/* HTTP/1.0)

A wiki with such setup could then disable the on-disk storage.


_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Roan Kattouw-2
In reply to this post by Tim Starling-2
On Sun, Sep 2, 2012 at 5:59 PM, Tim Starling <[hidden email]> wrote:
> The other reason for the existence of the backend thumbnail store is
> to transport images from the thumbnail scalers to the 404 handler. For
> that purpose, the image only needs to exist in the backend for a few
> seconds. It could be replaced by a better 404 handler, that sends
> thumbnails directly by HTTP. Maybe the Swift one does that already.
>
My understanding is that thumb.php already streamed the thumbnail back
to the 404 handler via HTTP and has done so for at least the past two
years or so.

Roan

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Jon Robson
I just wanted to clarify something... is there any protection in place in
the thumbnail generator to prevent denial of service attacks? For instance
if someone wanted to they could run a script which uploaded photos then
fired off requests for thumbnails of it of size 20px,21px,22px...1024px

I'm guessing the servers wouldn't like that. This is why I'd be keen to
limit the sizes.

May I suggest someone analyses the sizes currently used on wikipedia and we
limit to those as an initial step and then review the less frequently used
ones and standardise on some sizes?
On Sep 5, 2012 9:15 AM, "Roan Kattouw" <[hidden email]> wrote:

> On Sun, Sep 2, 2012 at 5:59 PM, Tim Starling <[hidden email]>
> wrote:
> > The other reason for the existence of the backend thumbnail store is
> > to transport images from the thumbnail scalers to the 404 handler. For
> > that purpose, the image only needs to exist in the backend for a few
> > seconds. It could be replaced by a better 404 handler, that sends
> > thumbnails directly by HTTP. Maybe the Swift one does that already.
> >
> My understanding is that thumb.php already streamed the thumbnail back
> to the 404 handler via HTTP and has done so for at least the past two
> years or so.
>
> Roan
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Asher Feldman
In reply to this post by Platonides
On Tue, Sep 4, 2012 at 3:11 PM, Platonides <[hidden email]> wrote:

> On 03/09/12 02:59, Tim Starling wrote:
> > I'll go for option 4. You can't delete the images from the backend
> > while they are still in Squid, because then they would not be purged
> > when the image is updated or action=purge is requested. In fact, that
> > is one of only two reasons for the existence of the backend thumbnail
> > store on Wikimedia. The thumbnail backend could be replaced by a text
> > file that stores a list of thumbnail filenames which were sent to
> > Squid within a window equivalent to the expiry time sent in the
> > Cache-Control header.
> >
> > The other reason for the existence of the backend thumbnail store is
> > to transport images from the thumbnail scalers to the 404 handler. For
> > that purpose, the image only needs to exist in the backend for a few
> > seconds. It could be replaced by a better 404 handler, that sends
> > thumbnails directly by HTTP. Maybe the Swift one does that already.
> >
> > -- Tim Starling
>
> The second one seems easy to fix. The first one should IMHO be fixed in
> squid/varnish by allowing wildcard purges (ie. PURGE
> /wikipedia/commons/thumb/5/5c/Tim_starling.jpg/* HTTP/1.0)
>

fast.ly  implements group purge for varnish like this via a proxy daemon
that watches backend responses for a "tag" response header (i.e. all
resolutions of Tim_starling.jpg would be tagged that) and builds an
in-memory hash of tags->objects which can be purged on.  I've been told
they'd probably open source the code for us if we want it, and it is
interesting (especially to deal with the fact that we don't purge articles
at all of their possible url's) albeit with its own challenges.  If we
implemented a backend system to track thumbnails that exist for a given
orig, we may be able to remove our dependency on swift container listings
to purge images, paving the way for a second class of thumbnails that are
only cached.

A wiki with such setup could then disable the on-disk storage.
>

I think this is entirely doable, but scaling the imagescalers to support
cache failures at wmf scale would be a waste, except perhaps for
non-standard sizes that aren't widely used.  I like Brion's thoughts on
revamping image handling, and would like to see semi-permanent (in swift)
storage of a standardized set of thumbnail resolutions but we could still
support additional resolutions.  Browser scaling is also at least worth
experimenting with.  Instances where browser scaling would be bad are
likely instances where the image is already subpar if viewed on a high-dpi
/ retina display.
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: scaled media (thumbs) as *temporary* files, not stored forever

Roan Kattouw-2
On Wed, Sep 5, 2012 at 12:35 PM, Asher Feldman <[hidden email]> wrote:
> Browser scaling is also at least worth
> experimenting with.  Instances where browser scaling would be bad are
> likely instances where the image is already subpar if viewed on a high-dpi
> / retina display.
Other instances where browser scaling is bad are:
* PoS browsers that don't render SVGs (how old are these by now?)
* Even modern browsers have subpar SVG rendering at 1x, PNG looks better
* Some media types are "scaled" in unusual ways (SVGs, but also video
stills, PDF pages, ...)
* Some original images are really friggin' large (20-30 megapixels
sometimes), so at least some downscaling is needed there
* Mobile clients will want to minimize the amount of data transferred

Roan

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
123