Video output changing to WebM VP9/Opus soon

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Video output changing to WebM VP9/Opus soon

Brion Vibber-4
In the next couple weeks I'm planning to start switching our video
transcode output from WebM VP8/Vorbis to the newer WebM VP9/Opus profile,
which saves us about 38% on file size and bandwidth while retaining the
same quality.

This will not affect what kinds of files you upload; only the scaled
transcoded output files used for playback will change. All modern browsers
that support VP8 support VP9 as well, and our player shim for Safari and IE
will continue to work.

All the details:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition

Comments and questions welcome!

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brian Wolff
Woo!

Good to see us moving forward to newer codecs.

What is the expected impact on transcode time/memory usage? (As in, will
large videos still transcode fine or would they potentially hit limits
sooner?)

--
brian

On Sunday, June 17, 2018, Brion Vibber <[hidden email]> wrote:
> In the next couple weeks I'm planning to start switching our video
transcode output from WebM VP8/Vorbis to the newer WebM VP9/Opus profile,
which saves us about 38% on file size and bandwidth while retaining the
same quality.
> This will not affect what kinds of files you upload; only the scaled
transcoded output files used for playback will change. All modern browsers
that support VP8 support VP9 as well, and our player shim for Safari and IE
will continue to work.
> All the details:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition
> Comments and questions welcome!
> -- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff <[hidden email]> wrote:

> Woo!
>
> Good to see us moving forward to newer codecs.
>

:D


> What is the expected impact on transcode time/memory usage? (As in, will
> large videos still transcode fine or would they potentially hit limits
> sooner?)
>

Using the same CPU thread count, typical files take about 3-4 times longer
with the configuration we've got ready. This doesn't seem to be a major
problem at low resolutions, but higher resolutions may need to have the
limits raised for very long videos like conference talks. Memory usage may
be higher but I've not specifically noticed a major difference between VP8
and VP9 there.

This could be improved significantly by updating to libvpx 1.7 and a
matching version of ffmpeg that supports macroblock row multithreading:
this means that we'll be able to use more than 4 threads for HD videos,
which should allow using all cores on a 20-core/40-thread machine if it's
not loaded up with other files.

The necessary libraries are released; we just need to backport the newer
packages to Debian 9 I believe. Don't yet know whether this will be an easy
or hard task.

Now that you mention it I am concerned about the time limit on very long
videos, so I'll take another look at the packaging backport and see if we
can knock that out quickly. :)

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Current state on this:

* delaying to mid-July so I don't start a batch process and leave it
unattended for a week :)
* still hoping to deploy the libvpx+ffmpeg backport first so we start with
best performance; Moritz made a start on libvpx but we still have to
resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to
3.4)
* after further testing of 30fps and 60fps files, I'm also planning to
increase the data rate a bit.

I want to do a little more testing on the ideal data rate, but it'll end up
looking more like a 30% reduction than a 38% reduction in file size. Our
current VP8 and provisional-VP9 configurations were tuned mostly on 24fps
files, while 30fps files are very common and require a moderately higher
data rate to reduce compression artifacts. (50fps and 60fps files are also
around, and will also benefit from an increase in data rate.)

(Possibly the data rate will end up scaled according to frame rate, but
frame rate is surprisingly hard to calculate reliably for WebM input.)

-- brion


On Sun, Jun 17, 2018 at 1:32 PM Brion Vibber <[hidden email]> wrote:

> On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff <[hidden email]> wrote:
>
>> Woo!
>>
>> Good to see us moving forward to newer codecs.
>>
>
> :D
>
>
>> What is the expected impact on transcode time/memory usage? (As in, will
>> large videos still transcode fine or would they potentially hit limits
>> sooner?)
>>
>
> Using the same CPU thread count, typical files take about 3-4 times longer
> with the configuration we've got ready. This doesn't seem to be a major
> problem at low resolutions, but higher resolutions may need to have the
> limits raised for very long videos like conference talks. Memory usage may
> be higher but I've not specifically noticed a major difference between VP8
> and VP9 there.
>
> This could be improved significantly by updating to libvpx 1.7 and a
> matching version of ffmpeg that supports macroblock row multithreading:
> this means that we'll be able to use more than 4 threads for HD videos,
> which should allow using all cores on a 20-core/40-thread machine if it's
> not loaded up with other files.
>
> The necessary libraries are released; we just need to backport the newer
> packages to Debian 9 I believe. Don't yet know whether this will be an easy
> or hard task.
>
> Now that you mention it I am concerned about the time limit on very long
> videos, so I'll take another look at the packaging backport and see if we
> can knock that out quickly. :)
>
> -- brion
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Pine W
Thanks for the update. As you can imagine, I am very interested in this!

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

-------- Original message --------From: Brion Vibber <[hidden email]> Date: 6/28/18  1:54 PM  (GMT-08:00) To: Wikimedia Foundation Multimedia Team <[hidden email]> Cc: Wikimedia-tech list <[hidden email]> Subject: Re: [Multimedia] Video output changing to WebM VP9/Opus soon
Current state on this:
* delaying to mid-July so I don't start a batch process and leave it unattended for a week :)* still hoping to deploy the libvpx+ffmpeg backport first so we start with best performance; Moritz made a start on libvpx but we still have to resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to 3.4)* after further testing of 30fps and 60fps files, I'm also planning to increase the data rate a bit.
I want to do a little more testing on the ideal data rate, but it'll end up looking more like a 30% reduction than a 38% reduction in file size. Our current VP8 and provisional-VP9 configurations were tuned mostly on 24fps files, while 30fps files are very common and require a moderately higher data rate to reduce compression artifacts. (50fps and 60fps files are also around, and will also benefit from an increase in data rate.)
(Possibly the data rate will end up scaled according to frame rate, but frame rate is surprisingly hard to calculate reliably for WebM input.)
-- brion

On Sun, Jun 17, 2018 at 1:32 PM Brion Vibber <[hidden email]> wrote:
On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff <[hidden email]> wrote:
Woo!

Good to see us moving forward to newer codecs.

:D What is the expected impact on transcode time/memory usage? (As in, will large videos still transcode fine or would they potentially hit limits sooner?)

Using the same CPU thread count, typical files take about 3-4 times longer with the configuration we've got ready. This doesn't seem to be a major problem at low resolutions, but higher resolutions may need to have the limits raised for very long videos like conference talks. Memory usage may be higher but I've not specifically noticed a major difference between VP8 and VP9 there.
This could be improved significantly by updating to libvpx 1.7 and a matching version of ffmpeg that supports macroblock row multithreading: this means that we'll be able to use more than 4 threads for HD videos, which should allow using all cores on a 20-core/40-thread machine if it's not loaded up with other files.
The necessary libraries are released; we just need to backport the newer packages to Debian 9 I believe. Don't yet know whether this will be an easy or hard task.
Now that you mention it I am concerned about the time limit on very long videos, so I'll take another look at the packaging backport and see if we can knock that out quickly. :)
-- brion

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Moritz Muehlenhoff
In reply to this post by Brion Vibber-4
Hi all,

On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> Current state on this:
>
> * still hoping to deploy the libvpx+ffmpeg backport first so we start with
> best performance; Moritz made a start on libvpx but we still have to
> resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to
> 3.4)

I've completed this today. We now have a separate repository component
for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
(thus allowing us to follow the ffmpeg security updates released in Debian
with a local rebuild) with backported row-mt support and linked against
libvpx 1.7.0.

I tested re-encoding
https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
(which is a nice fast-paced test file) from VP8 to VP9, which results in
a size reduction from 48M to 31M.

When using eight CPU cores on one of our video scaler servers, enabling row-mt
gives a significant performance boost; encoding time went down from 5:31 mins
to 3:36 mins.

All the details can be found at
https://phabricator.wikimedia.org/T190333#4324995

Cheers,
        Moritz

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Awesome sauce. Thanks Moritz!

-- brion

On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
[hidden email]> wrote:

> Hi all,
>
> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> > Current state on this:
> >
> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
> with
> > best performance; Moritz made a start on libvpx but we still have to
> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the way
> to
> > 3.4)
>
> I've completed this today. We now have a separate repository component
> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
> (thus allowing us to follow the ffmpeg security updates released in Debian
> with a local rebuild) with backported row-mt support and linked against
> libvpx 1.7.0.
>
> I tested re-encoding
>
> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
> (which is a nice fast-paced test file) from VP8 to VP9, which results in
> a size reduction from 48M to 31M.
>
> When using eight CPU cores on one of our video scaler servers, enabling
> row-mt
> gives a significant performance boost; encoding time went down from 5:31
> mins
> to 3:36 mins.
>
> All the details can be found at
> https://phabricator.wikimedia.org/T190333#4324995
>
> Cheers,
>         Moritz
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Ok, after some delay for re-tweaking the encoding settings for higher
quality when needed, and pulling in some other improvements to the config
system, all related updates to TimedMediaHandler have been merged. :D

If all goes well with the general deployments in the next few days, expect
the beginning of VP9 rollout starting next week.

Changes since the earlier announcement:
* the new row-multithreading will be available, which allows higher
threading usage at all resolutions; encoding times will be more like 1.5-2x
slower instead of 3-4x slower.
* switch to constrained quality with a larger max bitrate: many files will
become significantly smaller in their VP9 versions, but some will actually
increase in exchange for a huge increase in quality -- this is mostly 60fps
high-rate files, and those with lots of motion and detail that didn't
compress well at the default low data rates.

-- brion

On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]> wrote:

> Awesome sauce. Thanks Moritz!
>
> -- brion
>
> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
> [hidden email]> wrote:
>
>> Hi all,
>>
>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>> > Current state on this:
>> >
>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>> with
>> > best performance; Moritz made a start on libvpx but we still have to
>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>> way to
>> > 3.4)
>>
>> I've completed this today. We now have a separate repository component
>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>> (thus allowing us to follow the ffmpeg security updates released in Debian
>> with a local rebuild) with backported row-mt support and linked against
>> libvpx 1.7.0.
>>
>> I tested re-encoding
>>
>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>> a size reduction from 48M to 31M.
>>
>> When using eight CPU cores on one of our video scaler servers, enabling
>> row-mt
>> gives a significant performance boost; encoding time went down from 5:31
>> mins
>> to 3:36 mins.
>>
>> All the details can be found at
>> https://phabricator.wikimedia.org/T190333#4324995
>>
>> Cheers,
>>         Moritz
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Oh and one more thing!

For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
resolutions, which people can manually bump up to when watching videos with
a suitable 4K source on a high-res screen. They use higher data rates, but
only a small fraction of input files are 4K so should not significantly
increase disk space projections for now.

These can take a long time to compress, so if we find it's problematic
we'll turn them back off until the jobs can be split into tiny chunks
(future work planned!), but it works in my testing and shouldn't clog the
servers now that we have more available.

(Note that the ogv.js player shim for Safari will not handle
greater-than-HD resolutions fast enough for playback, even on a fast Mac or
iPad; for best results for 4K playback use Firefox, Chrome, or a
Chromium-based browser.)

-- brion

On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]> wrote:

> Ok, after some delay for re-tweaking the encoding settings for higher
> quality when needed, and pulling in some other improvements to the config
> system, all related updates to TimedMediaHandler have been merged. :D
>
> If all goes well with the general deployments in the next few days, expect
> the beginning of VP9 rollout starting next week.
>
> Changes since the earlier announcement:
> * the new row-multithreading will be available, which allows higher
> threading usage at all resolutions; encoding times will be more like 1.5-2x
> slower instead of 3-4x slower.
> * switch to constrained quality with a larger max bitrate: many files will
> become significantly smaller in their VP9 versions, but some will actually
> increase in exchange for a huge increase in quality -- this is mostly 60fps
> high-rate files, and those with lots of motion and detail that didn't
> compress well at the default low data rates.
>
> -- brion
>
> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
> wrote:
>
>> Awesome sauce. Thanks Moritz!
>>
>> -- brion
>>
>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>> [hidden email]> wrote:
>>
>>> Hi all,
>>>
>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>> > Current state on this:
>>> >
>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>>> with
>>> > best performance; Moritz made a start on libvpx but we still have to
>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>> way to
>>> > 3.4)
>>>
>>> I've completed this today. We now have a separate repository component
>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>>> (thus allowing us to follow the ffmpeg security updates released in
>>> Debian
>>> with a local rebuild) with backported row-mt support and linked against
>>> libvpx 1.7.0.
>>>
>>> I tested re-encoding
>>>
>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>>> a size reduction from 48M to 31M.
>>>
>>> When using eight CPU cores on one of our video scaler servers, enabling
>>> row-mt
>>> gives a significant performance boost; encoding time went down from 5:31
>>> mins
>>> to 3:36 mins.
>>>
>>> All the details can be found at
>>> https://phabricator.wikimedia.org/T190333#4324995
>>>
>>> Cheers,
>>>         Moritz
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list
>>> [hidden email]
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

rupert THURNER-2
i tried VP9 uploading a video with a spinning wheel made with a
samsung galaxy s5, mp4, original size 99.5MB. i removed the sound
beforehand though:
https://commons.wikimedia.org/wiki/File:180522-alpaca-spinnen-gotthard-passh%C3%B6he-silent.webm

using:
https://tools.wmflabs.org/video2commons/

which used the following (i removed the paths ...)
ffmpeg -y -i dl.unknown_video -threads 0 -skip_threshold 0 -bufsize
6000k -rc_init_occupancy 4000 -qmin 19 -qmax 19 -vcodec libvpx-vp9
-tile-columns 4 -f webm -ss 0 -an -pass 1 -passlogfile
dl.unknown_video.an.vp9.webm.log /dev/null

ffmpeg -y -i dl.unknown_video -threads 0 -skip_threshold 0 -bufsize
6000k -rc_init_occupancy 4000 -qmin 19 -qmax 19 -vcodec libvpx-vp9
-tile-columns 4 -f webm -ss 0 -an -pass 2 -passlogfile
dl.unknown_video.an.vp9.webm.log dl.unknown_video.an.vp9.webm

the size increased by 30 mb, the quality on a smaller screen is the
same, i did not verify on a high resolution screen if a difference can
be noticed.

rupert


On Fri, Jul 27, 2018 at 2:47 AM, Brion Vibber <[hidden email]> wrote:

> Oh and one more thing!
>
> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
> resolutions, which people can manually bump up to when watching videos with
> a suitable 4K source on a high-res screen. They use higher data rates, but
> only a small fraction of input files are 4K so should not significantly
> increase disk space projections for now.
>
> These can take a long time to compress, so if we find it's problematic we'll
> turn them back off until the jobs can be split into tiny chunks (future work
> planned!), but it works in my testing and shouldn't clog the servers now
> that we have more available.
>
> (Note that the ogv.js player shim for Safari will not handle greater-than-HD
> resolutions fast enough for playback, even on a fast Mac or iPad; for best
> results for 4K playback use Firefox, Chrome, or a Chromium-based browser.)
>
> -- brion
>
> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]> wrote:
>>
>> Ok, after some delay for re-tweaking the encoding settings for higher
>> quality when needed, and pulling in some other improvements to the config
>> system, all related updates to TimedMediaHandler have been merged. :D
>>
>> If all goes well with the general deployments in the next few days, expect
>> the beginning of VP9 rollout starting next week.
>>
>> Changes since the earlier announcement:
>> * the new row-multithreading will be available, which allows higher
>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>> slower instead of 3-4x slower.
>> * switch to constrained quality with a larger max bitrate: many files will
>> become significantly smaller in their VP9 versions, but some will actually
>> increase in exchange for a huge increase in quality -- this is mostly 60fps
>> high-rate files, and those with lots of motion and detail that didn't
>> compress well at the default low data rates.
>>
>> -- brion
>>
>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>> wrote:
>>>
>>> Awesome sauce. Thanks Moritz!
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff
>>> <[hidden email]> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>> > Current state on this:
>>>> >
>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>>>> > with
>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>>> > way to
>>>> > 3.4)
>>>>
>>>> I've completed this today. We now have a separate repository component
>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>> Debian
>>>> with a local rebuild) with backported row-mt support and linked against
>>>> libvpx 1.7.0.
>>>>
>>>> I tested re-encoding
>>>>
>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>>>> a size reduction from 48M to 31M.
>>>>
>>>> When using eight CPU cores on one of our video scaler servers, enabling
>>>> row-mt
>>>> gives a significant performance boost; encoding time went down from 5:31
>>>> mins
>>>> to 3:36 mins.
>>>>
>>>> All the details can be found at
>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>
>>>> Cheers,
>>>>         Moritz
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list
>>>> [hidden email]
>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
> _______________________________________________
> Multimedia mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/multimedia
>

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
The quality/bandwidth for the output in this command will be determined by
the qmin/qmax settings you provided; the value of 19 is a bit conservative
and will likely produce a relatively large, but good quality file. Try
various different settings to dial in on desired quality; larger settings
for qmin/qmax will be lower quality / smaller bandwidth.

The settings we'll be using for derivatives are based on the constrained
quality mode, using the -crf option plus a relatively high maximum bitrate
as a constraint for high-motion files, and also a conservative -qmin to
ensure that bits aren't wasted on details that are actually source
compression artifacts.


In general, you should optimize the files you upload for quality, not for
size -- higher source quality ensures better quality for the viewable
derivatives, and for future re-use possibilities. Note that you can keep
using VP8 for uploaded files; there's no immediate need to change
configurations for uploads unless the files themselves are too big to
upload and need higher compression.

-- brion

On Thu, Jul 26, 2018 at 9:38 PM rupert THURNER <[hidden email]>
wrote:

> i tried VP9 uploading a video with a spinning wheel made with a
> samsung galaxy s5, mp4, original size 99.5MB. i removed the sound
> beforehand though:
>
> https://commons.wikimedia.org/wiki/File:180522-alpaca-spinnen-gotthard-passh%C3%B6he-silent.webm
>
> using:
> https://tools.wmflabs.org/video2commons/
>
> which used the following (i removed the paths ...)
> ffmpeg -y -i dl.unknown_video -threads 0 -skip_threshold 0 -bufsize
> 6000k -rc_init_occupancy 4000 -qmin 19 -qmax 19 -vcodec libvpx-vp9
> -tile-columns 4 -f webm -ss 0 -an -pass 1 -passlogfile
> dl.unknown_video.an.vp9.webm.log /dev/null
>
> ffmpeg -y -i dl.unknown_video -threads 0 -skip_threshold 0 -bufsize
> 6000k -rc_init_occupancy 4000 -qmin 19 -qmax 19 -vcodec libvpx-vp9
> -tile-columns 4 -f webm -ss 0 -an -pass 2 -passlogfile
> dl.unknown_video.an.vp9.webm.log dl.unknown_video.an.vp9.webm
>
> the size increased by 30 mb, the quality on a smaller screen is the
> same, i did not verify on a high resolution screen if a difference can
> be noticed.
>
> rupert
>
>
> On Fri, Jul 27, 2018 at 2:47 AM, Brion Vibber <[hidden email]>
> wrote:
> > Oh and one more thing!
> >
> > For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
> > resolutions, which people can manually bump up to when watching videos
> with
> > a suitable 4K source on a high-res screen. They use higher data rates,
> but
> > only a small fraction of input files are 4K so should not significantly
> > increase disk space projections for now.
> >
> > These can take a long time to compress, so if we find it's problematic
> we'll
> > turn them back off until the jobs can be split into tiny chunks (future
> work
> > planned!), but it works in my testing and shouldn't clog the servers now
> > that we have more available.
> >
> > (Note that the ogv.js player shim for Safari will not handle
> greater-than-HD
> > resolutions fast enough for playback, even on a fast Mac or iPad; for
> best
> > results for 4K playback use Firefox, Chrome, or a Chromium-based
> browser.)
> >
> > -- brion
> >
> > On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
> wrote:
> >>
> >> Ok, after some delay for re-tweaking the encoding settings for higher
> >> quality when needed, and pulling in some other improvements to the
> config
> >> system, all related updates to TimedMediaHandler have been merged. :D
> >>
> >> If all goes well with the general deployments in the next few days,
> expect
> >> the beginning of VP9 rollout starting next week.
> >>
> >> Changes since the earlier announcement:
> >> * the new row-multithreading will be available, which allows higher
> >> threading usage at all resolutions; encoding times will be more like
> 1.5-2x
> >> slower instead of 3-4x slower.
> >> * switch to constrained quality with a larger max bitrate: many files
> will
> >> become significantly smaller in their VP9 versions, but some will
> actually
> >> increase in exchange for a huge increase in quality -- this is mostly
> 60fps
> >> high-rate files, and those with lots of motion and detail that didn't
> >> compress well at the default low data rates.
> >>
> >> -- brion
> >>
> >> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
> >> wrote:
> >>>
> >>> Awesome sauce. Thanks Moritz!
> >>>
> >>> -- brion
> >>>
> >>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff
> >>> <[hidden email]> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> >>>> > Current state on this:
> >>>> >
> >>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
> start
> >>>> > with
> >>>> > best performance; Moritz made a start on libvpx but we still have to
> >>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
> >>>> > way to
> >>>> > 3.4)
> >>>>
> >>>> I've completed this today. We now have a separate repository component
> >>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
> 3.2.10
> >>>> (thus allowing us to follow the ffmpeg security updates released in
> >>>> Debian
> >>>> with a local rebuild) with backported row-mt support and linked
> against
> >>>> libvpx 1.7.0.
> >>>>
> >>>> I tested re-encoding
> >>>>
> >>>>
> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
> >>>> (which is a nice fast-paced test file) from VP8 to VP9, which results
> in
> >>>> a size reduction from 48M to 31M.
> >>>>
> >>>> When using eight CPU cores on one of our video scaler servers,
> enabling
> >>>> row-mt
> >>>> gives a significant performance boost; encoding time went down from
> 5:31
> >>>> mins
> >>>> to 3:36 mins.
> >>>>
> >>>> All the details can be found at
> >>>> https://phabricator.wikimedia.org/T190333#4324995
> >>>>
> >>>> Cheers,
> >>>>         Moritz
> >>>>
> >>>> _______________________________________________
> >>>> Wikitech-l mailing list
> >>>> [hidden email]
> >>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> >
> > _______________________________________________
> > Multimedia mailing list
> > [hidden email]
> > https://lists.wikimedia.org/mailman/listinfo/multimedia
> >
>
> _______________________________________________
> Multimedia mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/multimedia
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
In reply to this post by Brion Vibber-4
Configuration has been updated and VP9 mode swapped in. Newly uploaded
videos should start encoding as VP9 starting now.

I'll start the backfill batch process tonight or tomorrow, and will likely
stop and restart it a few times over the coming weeks if anything needs
adjustment. Please file tasks in phabricator and/or give me a ping on IRC
if any issues come up with the new conversions, or with old files!
(Existing VP8 files will be left as-is for now until we're sure
everything's up to snuff.)

Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
deployments, with the patches and review, and with final deployment which
always takes longer than you expect. :) This'll be a nice improvement to
our video output for now, and lays a lot of groundwork for next steps.

-- brion


On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]> wrote:

> Oh and one more thing!
>
> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
> resolutions, which people can manually bump up to when watching videos with
> a suitable 4K source on a high-res screen. They use higher data rates, but
> only a small fraction of input files are 4K so should not significantly
> increase disk space projections for now.
>
> These can take a long time to compress, so if we find it's problematic
> we'll turn them back off until the jobs can be split into tiny chunks
> (future work planned!), but it works in my testing and shouldn't clog the
> servers now that we have more available.
>
> (Note that the ogv.js player shim for Safari will not handle
> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
> iPad; for best results for 4K playback use Firefox, Chrome, or a
> Chromium-based browser.)
>
> -- brion
>
> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
> wrote:
>
>> Ok, after some delay for re-tweaking the encoding settings for higher
>> quality when needed, and pulling in some other improvements to the config
>> system, all related updates to TimedMediaHandler have been merged. :D
>>
>> If all goes well with the general deployments in the next few days,
>> expect the beginning of VP9 rollout starting next week.
>>
>> Changes since the earlier announcement:
>> * the new row-multithreading will be available, which allows higher
>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>> slower instead of 3-4x slower.
>> * switch to constrained quality with a larger max bitrate: many files
>> will become significantly smaller in their VP9 versions, but some will
>> actually increase in exchange for a huge increase in quality -- this is
>> mostly 60fps high-rate files, and those with lots of motion and detail that
>> didn't compress well at the default low data rates.
>>
>> -- brion
>>
>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Awesome sauce. Thanks Moritz!
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>> [hidden email]> wrote:
>>>
>>>> Hi all,
>>>>
>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>> > Current state on this:
>>>> >
>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>>>> with
>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>>> way to
>>>> > 3.4)
>>>>
>>>> I've completed this today. We now have a separate repository component
>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>> Debian
>>>> with a local rebuild) with backported row-mt support and linked against
>>>> libvpx 1.7.0.
>>>>
>>>> I tested re-encoding
>>>>
>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>>>> a size reduction from 48M to 31M.
>>>>
>>>> When using eight CPU cores on one of our video scaler servers, enabling
>>>> row-mt
>>>> gives a significant performance boost; encoding time went down from
>>>> 5:31 mins
>>>> to 3:36 mins.
>>>>
>>>> All the details can be found at
>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>
>>>> Cheers,
>>>>         Moritz
>>>>
>>>> _______________________________________________
>>>> Wikitech-l mailing list
>>>> [hidden email]
>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>
>>>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Pine W
Thanks, Brion. At my current rate of progress, I'll be making use of this
feature by the end of the month.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

On Tue, Jul 31, 2018 at 12:51 AM, Brion Vibber <[hidden email]>
wrote:

> Configuration has been updated and VP9 mode swapped in. Newly uploaded
> videos should start encoding as VP9 starting now.
>
> I'll start the backfill batch process tonight or tomorrow, and will likely
> stop and restart it a few times over the coming weeks if anything needs
> adjustment. Please file tasks in phabricator and/or give me a ping on IRC
> if any issues come up with the new conversions, or with old files!
> (Existing VP8 files will be left as-is for now until we're sure
> everything's up to snuff.)
>
> Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
> deployments, with the patches and review, and with final deployment which
> always takes longer than you expect. :) This'll be a nice improvement to
> our video output for now, and lays a lot of groundwork for next steps.
>
> -- brion
>
>
> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]>
> wrote:
>
>> Oh and one more thing!
>>
>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>> resolutions, which people can manually bump up to when watching videos with
>> a suitable 4K source on a high-res screen. They use higher data rates, but
>> only a small fraction of input files are 4K so should not significantly
>> increase disk space projections for now.
>>
>> These can take a long time to compress, so if we find it's problematic
>> we'll turn them back off until the jobs can be split into tiny chunks
>> (future work planned!), but it works in my testing and shouldn't clog the
>> servers now that we have more available.
>>
>> (Note that the ogv.js player shim for Safari will not handle
>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>> Chromium-based browser.)
>>
>> -- brion
>>
>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>> quality when needed, and pulling in some other improvements to the config
>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>
>>> If all goes well with the general deployments in the next few days,
>>> expect the beginning of VP9 rollout starting next week.
>>>
>>> Changes since the earlier announcement:
>>> * the new row-multithreading will be available, which allows higher
>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>> slower instead of 3-4x slower.
>>> * switch to constrained quality with a larger max bitrate: many files
>>> will become significantly smaller in their VP9 versions, but some will
>>> actually increase in exchange for a huge increase in quality -- this is
>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>> didn't compress well at the default low data rates.
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>>> wrote:
>>>
>>>> Awesome sauce. Thanks Moritz!
>>>>
>>>> -- brion
>>>>
>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>> [hidden email]> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>> > Current state on this:
>>>>> >
>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>>>>> start with
>>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>>>> way to
>>>>> > 3.4)
>>>>>
>>>>> I've completed this today. We now have a separate repository component
>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>>>>> 3.2.10
>>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>>> Debian
>>>>> with a local rebuild) with backported row-mt support and linked against
>>>>> libvpx 1.7.0.
>>>>>
>>>>> I tested re-encoding
>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_
>>>>> Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results
>>>>> in
>>>>> a size reduction from 48M to 31M.
>>>>>
>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>> enabling row-mt
>>>>> gives a significant performance boost; encoding time went down from
>>>>> 5:31 mins
>>>>> to 3:36 mins.
>>>>>
>>>>> All the details can be found at
>>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>>
>>>>> Cheers,
>>>>>         Moritz
>>>>>
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list
>>>>> [hidden email]
>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>
>>>>
> _______________________________________________
> Multimedia mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/multimedia
>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Pine W
I meant end of August. Sorry for the extra email noise.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

On Tue, Jul 31, 2018 at 2:15 AM, Pine W <[hidden email]> wrote:

> Thanks, Brion. At my current rate of progress, I'll be making use of this
> feature by the end of the month.
>
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
> On Tue, Jul 31, 2018 at 12:51 AM, Brion Vibber <[hidden email]>
> wrote:
>
>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>> videos should start encoding as VP9 starting now.
>>
>> I'll start the backfill batch process tonight or tomorrow, and will
>> likely stop and restart it a few times over the coming weeks if anything
>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>> IRC if any issues come up with the new conversions, or with old files!
>> (Existing VP8 files will be left as-is for now until we're sure
>> everything's up to snuff.)
>>
>> Big thanks to everybody who's helped prepping this, with libvpx and
>> ffmpeg deployments, with the patches and review, and with final deployment
>> which always takes longer than you expect. :) This'll be a nice improvement
>> to our video output for now, and lays a lot of groundwork for next steps.
>>
>> -- brion
>>
>>
>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Oh and one more thing!
>>>
>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>> resolutions, which people can manually bump up to when watching videos with
>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>> only a small fraction of input files are 4K so should not significantly
>>> increase disk space projections for now.
>>>
>>> These can take a long time to compress, so if we find it's problematic
>>> we'll turn them back off until the jobs can be split into tiny chunks
>>> (future work planned!), but it works in my testing and shouldn't clog the
>>> servers now that we have more available.
>>>
>>> (Note that the ogv.js player shim for Safari will not handle
>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>> Chromium-based browser.)
>>>
>>> -- brion
>>>
>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
>>> wrote:
>>>
>>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>>> quality when needed, and pulling in some other improvements to the config
>>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>>
>>>> If all goes well with the general deployments in the next few days,
>>>> expect the beginning of VP9 rollout starting next week.
>>>>
>>>> Changes since the earlier announcement:
>>>> * the new row-multithreading will be available, which allows higher
>>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>>> slower instead of 3-4x slower.
>>>> * switch to constrained quality with a larger max bitrate: many files
>>>> will become significantly smaller in their VP9 versions, but some will
>>>> actually increase in exchange for a huge increase in quality -- this is
>>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>>> didn't compress well at the default low data rates.
>>>>
>>>> -- brion
>>>>
>>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>>>> wrote:
>>>>
>>>>> Awesome sauce. Thanks Moritz!
>>>>>
>>>>> -- brion
>>>>>
>>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>>> > Current state on this:
>>>>>> >
>>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>>>>>> start with
>>>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all
>>>>>> the way to
>>>>>> > 3.4)
>>>>>>
>>>>>> I've completed this today. We now have a separate repository component
>>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>>>>>> 3.2.10
>>>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>>>> Debian
>>>>>> with a local rebuild) with backported row-mt support and linked
>>>>>> against
>>>>>> libvpx 1.7.0.
>>>>>>
>>>>>> I tested re-encoding
>>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitt
>>>>>> s_Todeswand_2017_-_Jagath_Perera.webm
>>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results
>>>>>> in
>>>>>> a size reduction from 48M to 31M.
>>>>>>
>>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>>> enabling row-mt
>>>>>> gives a significant performance boost; encoding time went down from
>>>>>> 5:31 mins
>>>>>> to 3:36 mins.
>>>>>>
>>>>>> All the details can be found at
>>>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>>>
>>>>>> Cheers,
>>>>>>         Moritz
>>>>>>
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list
>>>>>> [hidden email]
>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>>
>>>>>
>> _______________________________________________
>> Multimedia mailing list
>> [hidden email]
>> https://lists.wikimedia.org/mailman/listinfo/multimedia
>>
>>
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
In reply to this post by Brion Vibber-4
Batch process is continuing... over 2200 source videos have been compressed
to VP9 so far, of the 120k+ total in the system.

Seeing big improvements in overall compression, though it's hard to tell
how representative the subset of conversions done so far is. I'll post
occasional updates to the transcode report charts at:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports

In the histograms you can see that not only is the average bitrate down for
VP9 vs VP8, but there's a larger "spread" between low and high bandwidth in
the VP9 versions thanks to the constrained-quality configuration vs the old
fixed bitrate target. This allows files that compress well to use fewer
bits, while those that have high frame rates or lots of detail/motion use
more bits to encode higher quality than they did before.

It will take at least a few weeks to recompress everything, and we may or
may not want to increase the number of simultaneous jobs (so far the
servers are running beautifully, but are a bit under-loaded).

-- brion


On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber <[hidden email]> wrote:

> Configuration has been updated and VP9 mode swapped in. Newly uploaded
> videos should start encoding as VP9 starting now.
>
> I'll start the backfill batch process tonight or tomorrow, and will likely
> stop and restart it a few times over the coming weeks if anything needs
> adjustment. Please file tasks in phabricator and/or give me a ping on IRC
> if any issues come up with the new conversions, or with old files!
> (Existing VP8 files will be left as-is for now until we're sure
> everything's up to snuff.)
>
> Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
> deployments, with the patches and review, and with final deployment which
> always takes longer than you expect. :) This'll be a nice improvement to
> our video output for now, and lays a lot of groundwork for next steps.
>
> -- brion
>
>
> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]>
> wrote:
>
>> Oh and one more thing!
>>
>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>> resolutions, which people can manually bump up to when watching videos with
>> a suitable 4K source on a high-res screen. They use higher data rates, but
>> only a small fraction of input files are 4K so should not significantly
>> increase disk space projections for now.
>>
>> These can take a long time to compress, so if we find it's problematic
>> we'll turn them back off until the jobs can be split into tiny chunks
>> (future work planned!), but it works in my testing and shouldn't clog the
>> servers now that we have more available.
>>
>> (Note that the ogv.js player shim for Safari will not handle
>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>> Chromium-based browser.)
>>
>> -- brion
>>
>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>> quality when needed, and pulling in some other improvements to the config
>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>
>>> If all goes well with the general deployments in the next few days,
>>> expect the beginning of VP9 rollout starting next week.
>>>
>>> Changes since the earlier announcement:
>>> * the new row-multithreading will be available, which allows higher
>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>> slower instead of 3-4x slower.
>>> * switch to constrained quality with a larger max bitrate: many files
>>> will become significantly smaller in their VP9 versions, but some will
>>> actually increase in exchange for a huge increase in quality -- this is
>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>> didn't compress well at the default low data rates.
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>>> wrote:
>>>
>>>> Awesome sauce. Thanks Moritz!
>>>>
>>>> -- brion
>>>>
>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>> [hidden email]> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>> > Current state on this:
>>>>> >
>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>>>>> start with
>>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>>>> way to
>>>>> > 3.4)
>>>>>
>>>>> I've completed this today. We now have a separate repository component
>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>>>>> 3.2.10
>>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>>> Debian
>>>>> with a local rebuild) with backported row-mt support and linked against
>>>>> libvpx 1.7.0.
>>>>>
>>>>> I tested re-encoding
>>>>>
>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results
>>>>> in
>>>>> a size reduction from 48M to 31M.
>>>>>
>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>> enabling row-mt
>>>>> gives a significant performance boost; encoding time went down from
>>>>> 5:31 mins
>>>>> to 3:36 mins.
>>>>>
>>>>> All the details can be found at
>>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>>
>>>>> Cheers,
>>>>>         Moritz
>>>>>
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list
>>>>> [hidden email]
>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>
>>>>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Quick update... no major problems noticed so far. One configuration change
I made was to re-enable VP8 transcodes for new files, since disabling them
completely lead to the existing ones not being used. Later in the
transition, or afterwards, we'll clean up the now-unused VP8 transcodes.

If I'm reading numbers in
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
transition/reports right, almost 5% of files by count or 12% by duration
have been converted so far.

That leads to a remaining duration somewhere between 9 weeks (assuming 9
days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
actual factors like whether the remaining files are the same mix of
resolutions, and how we adjust the load balance on the job queue.

-- brion

On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber <[hidden email]> wrote:

> Batch process is continuing... over 2200 source videos have been
> compressed to VP9 so far, of the 120k+ total in the system.
>
> Seeing big improvements in overall compression, though it's hard to tell
> how representative the subset of conversions done so far is. I'll post
> occasional updates to the transcode report charts at:
>
> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports
>
> In the histograms you can see that not only is the average bitrate down
> for VP9 vs VP8, but there's a larger "spread" between low and high
> bandwidth in the VP9 versions thanks to the constrained-quality
> configuration vs the old fixed bitrate target. This allows files that
> compress well to use fewer bits, while those that have high frame rates or
> lots of detail/motion use more bits to encode higher quality than they did
> before.
>
> It will take at least a few weeks to recompress everything, and we may or
> may not want to increase the number of simultaneous jobs (so far the
> servers are running beautifully, but are a bit under-loaded).
>
> -- brion
>
>
> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber <[hidden email]>
> wrote:
>
>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>> videos should start encoding as VP9 starting now.
>>
>> I'll start the backfill batch process tonight or tomorrow, and will
>> likely stop and restart it a few times over the coming weeks if anything
>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>> IRC if any issues come up with the new conversions, or with old files!
>> (Existing VP8 files will be left as-is for now until we're sure
>> everything's up to snuff.)
>>
>> Big thanks to everybody who's helped prepping this, with libvpx and
>> ffmpeg deployments, with the patches and review, and with final deployment
>> which always takes longer than you expect. :) This'll be a nice improvement
>> to our video output for now, and lays a lot of groundwork for next steps.
>>
>> -- brion
>>
>>
>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Oh and one more thing!
>>>
>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>> resolutions, which people can manually bump up to when watching videos with
>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>> only a small fraction of input files are 4K so should not significantly
>>> increase disk space projections for now.
>>>
>>> These can take a long time to compress, so if we find it's problematic
>>> we'll turn them back off until the jobs can be split into tiny chunks
>>> (future work planned!), but it works in my testing and shouldn't clog the
>>> servers now that we have more available.
>>>
>>> (Note that the ogv.js player shim for Safari will not handle
>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>> Chromium-based browser.)
>>>
>>> -- brion
>>>
>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
>>> wrote:
>>>
>>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>>> quality when needed, and pulling in some other improvements to the config
>>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>>
>>>> If all goes well with the general deployments in the next few days,
>>>> expect the beginning of VP9 rollout starting next week.
>>>>
>>>> Changes since the earlier announcement:
>>>> * the new row-multithreading will be available, which allows higher
>>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>>> slower instead of 3-4x slower.
>>>> * switch to constrained quality with a larger max bitrate: many files
>>>> will become significantly smaller in their VP9 versions, but some will
>>>> actually increase in exchange for a huge increase in quality -- this is
>>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>>> didn't compress well at the default low data rates.
>>>>
>>>> -- brion
>>>>
>>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>>>> wrote:
>>>>
>>>>> Awesome sauce. Thanks Moritz!
>>>>>
>>>>> -- brion
>>>>>
>>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>>> > Current state on this:
>>>>>> >
>>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>>>>>> start with
>>>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all
>>>>>> the way to
>>>>>> > 3.4)
>>>>>>
>>>>>> I've completed this today. We now have a separate repository component
>>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>>>>>> 3.2.10
>>>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>>>> Debian
>>>>>> with a local rebuild) with backported row-mt support and linked
>>>>>> against
>>>>>> libvpx 1.7.0.
>>>>>>
>>>>>> I tested re-encoding
>>>>>>
>>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results
>>>>>> in
>>>>>> a size reduction from 48M to 31M.
>>>>>>
>>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>>> enabling row-mt
>>>>>> gives a significant performance boost; encoding time went down from
>>>>>> 5:31 mins
>>>>>> to 3:36 mins.
>>>>>>
>>>>>> All the details can be found at
>>>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>>>
>>>>>> Cheers,
>>>>>>         Moritz
>>>>>>
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list
>>>>>> [hidden email]
>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>>
>>>>>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
These conversions have been running in the background, at a moderate load
to ensure we don't slow down newer uploads much, and are up to the "P"s
alphabetically so we're on schedule. :)

I've temporarily stopped the batch process pending the upcoming datacenter
switchover test, and will continue once it's switched.

-- brion

On Wed, Aug 8, 2018 at 2:06 PM Brion Vibber <[hidden email]> wrote:

> Quick update... no major problems noticed so far. One configuration change
> I made was to re-enable VP8 transcodes for new files, since disabling them
> completely lead to the existing ones not being used. Later in the
> transition, or afterwards, we'll clean up the now-unused VP8 transcodes.
>
> If I'm reading numbers in
> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
> transition/reports right, almost 5% of files by count or 12% by duration
> have been converted so far.
>
> That leads to a remaining duration somewhere between 9 weeks (assuming 9
> days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
> actual factors like whether the remaining files are the same mix of
> resolutions, and how we adjust the load balance on the job queue.
>
> -- brion
>
> On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber <[hidden email]>
> wrote:
>
>> Batch process is continuing... over 2200 source videos have been
>> compressed to VP9 so far, of the 120k+ total in the system.
>>
>> Seeing big improvements in overall compression, though it's hard to tell
>> how representative the subset of conversions done so far is. I'll post
>> occasional updates to the transcode report charts at:
>>
>> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports
>>
>> In the histograms you can see that not only is the average bitrate down
>> for VP9 vs VP8, but there's a larger "spread" between low and high
>> bandwidth in the VP9 versions thanks to the constrained-quality
>> configuration vs the old fixed bitrate target. This allows files that
>> compress well to use fewer bits, while those that have high frame rates or
>> lots of detail/motion use more bits to encode higher quality than they did
>> before.
>>
>> It will take at least a few weeks to recompress everything, and we may or
>> may not want to increase the number of simultaneous jobs (so far the
>> servers are running beautifully, but are a bit under-loaded).
>>
>> -- brion
>>
>>
>> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>>> videos should start encoding as VP9 starting now.
>>>
>>> I'll start the backfill batch process tonight or tomorrow, and will
>>> likely stop and restart it a few times over the coming weeks if anything
>>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>>> IRC if any issues come up with the new conversions, or with old files!
>>> (Existing VP8 files will be left as-is for now until we're sure
>>> everything's up to snuff.)
>>>
>>> Big thanks to everybody who's helped prepping this, with libvpx and
>>> ffmpeg deployments, with the patches and review, and with final deployment
>>> which always takes longer than you expect. :) This'll be a nice improvement
>>> to our video output for now, and lays a lot of groundwork for next steps.
>>>
>>> -- brion
>>>
>>>
>>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]>
>>> wrote:
>>>
>>>> Oh and one more thing!
>>>>
>>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>>> resolutions, which people can manually bump up to when watching videos with
>>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>>> only a small fraction of input files are 4K so should not significantly
>>>> increase disk space projections for now.
>>>>
>>>> These can take a long time to compress, so if we find it's problematic
>>>> we'll turn them back off until the jobs can be split into tiny chunks
>>>> (future work planned!), but it works in my testing and shouldn't clog the
>>>> servers now that we have more available.
>>>>
>>>> (Note that the ogv.js player shim for Safari will not handle
>>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>>> Chromium-based browser.)
>>>>
>>>> -- brion
>>>>
>>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
>>>> wrote:
>>>>
>>>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>>>> quality when needed, and pulling in some other improvements to the config
>>>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>>>
>>>>> If all goes well with the general deployments in the next few days,
>>>>> expect the beginning of VP9 rollout starting next week.
>>>>>
>>>>> Changes since the earlier announcement:
>>>>> * the new row-multithreading will be available, which allows higher
>>>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>>>> slower instead of 3-4x slower.
>>>>> * switch to constrained quality with a larger max bitrate: many files
>>>>> will become significantly smaller in their VP9 versions, but some will
>>>>> actually increase in exchange for a huge increase in quality -- this is
>>>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>>>> didn't compress well at the default low data rates.
>>>>>
>>>>> -- brion
>>>>>
>>>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Awesome sauce. Thanks Moritz!
>>>>>>
>>>>>> -- brion
>>>>>>
>>>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>>>> > Current state on this:
>>>>>>> >
>>>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>>>>>>> start with
>>>>>>> > best performance; Moritz made a start on libvpx but we still have
>>>>>>> to
>>>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all
>>>>>>> the way to
>>>>>>> > 3.4)
>>>>>>>
>>>>>>> I've completed this today. We now have a separate repository
>>>>>>> component
>>>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>>>>>>> 3.2.10
>>>>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>>>>> Debian
>>>>>>> with a local rebuild) with backported row-mt support and linked
>>>>>>> against
>>>>>>> libvpx 1.7.0.
>>>>>>>
>>>>>>> I tested re-encoding
>>>>>>>
>>>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which
>>>>>>> results in
>>>>>>> a size reduction from 48M to 31M.
>>>>>>>
>>>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>>>> enabling row-mt
>>>>>>> gives a significant performance boost; encoding time went down from
>>>>>>> 5:31 mins
>>>>>>> to 3:36 mins.
>>>>>>>
>>>>>>> All the details can be found at
>>>>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>>>>
>>>>>>> Cheers,
>>>>>>>         Moritz
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Wikitech-l mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>>>
>>>>>>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Just an update -- background transcoding has been restarted now that we're
switched back to the Virginia data center where the majority of encoders
are.

Converting the lat 1/3 of videos to VP9 should be finished by the end of
November, approximately. At that point we'll probably start removing the
old VP8 versions to free up space.

-- brion

On Mon, Sep 10, 2018 at 12:05 PM Brion Vibber <[hidden email]> wrote:

> These conversions have been running in the background, at a moderate load
> to ensure we don't slow down newer uploads much, and are up to the "P"s
> alphabetically so we're on schedule. :)
>
> I've temporarily stopped the batch process pending the upcoming datacenter
> switchover test, and will continue once it's switched.
>
> -- brion
>
> On Wed, Aug 8, 2018 at 2:06 PM Brion Vibber <[hidden email]> wrote:
>
>> Quick update... no major problems noticed so far. One configuration
>> change I made was to re-enable VP8 transcodes for new files, since
>> disabling them completely lead to the existing ones not being used. Later
>> in the transition, or afterwards, we'll clean up the now-unused VP8
>> transcodes.
>>
>> If I'm reading numbers in
>> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
>> transition/reports right, almost 5% of files by count or 12% by duration
>> have been converted so far.
>>
>> That leads to a remaining duration somewhere between 9 weeks (assuming 9
>> days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
>> actual factors like whether the remaining files are the same mix of
>> resolutions, and how we adjust the load balance on the job queue.
>>
>> -- brion
>>
>> On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber <[hidden email]>
>> wrote:
>>
>>> Batch process is continuing... over 2200 source videos have been
>>> compressed to VP9 so far, of the 120k+ total in the system.
>>>
>>> Seeing big improvements in overall compression, though it's hard to tell
>>> how representative the subset of conversions done so far is. I'll post
>>> occasional updates to the transcode report charts at:
>>>
>>> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports
>>>
>>> In the histograms you can see that not only is the average bitrate down
>>> for VP9 vs VP8, but there's a larger "spread" between low and high
>>> bandwidth in the VP9 versions thanks to the constrained-quality
>>> configuration vs the old fixed bitrate target. This allows files that
>>> compress well to use fewer bits, while those that have high frame rates or
>>> lots of detail/motion use more bits to encode higher quality than they did
>>> before.
>>>
>>> It will take at least a few weeks to recompress everything, and we may
>>> or may not want to increase the number of simultaneous jobs (so far the
>>> servers are running beautifully, but are a bit under-loaded).
>>>
>>> -- brion
>>>
>>>
>>> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber <[hidden email]>
>>> wrote:
>>>
>>>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>>>> videos should start encoding as VP9 starting now.
>>>>
>>>> I'll start the backfill batch process tonight or tomorrow, and will
>>>> likely stop and restart it a few times over the coming weeks if anything
>>>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>>>> IRC if any issues come up with the new conversions, or with old files!
>>>> (Existing VP8 files will be left as-is for now until we're sure
>>>> everything's up to snuff.)
>>>>
>>>> Big thanks to everybody who's helped prepping this, with libvpx and
>>>> ffmpeg deployments, with the patches and review, and with final deployment
>>>> which always takes longer than you expect. :) This'll be a nice improvement
>>>> to our video output for now, and lays a lot of groundwork for next steps.
>>>>
>>>> -- brion
>>>>
>>>>
>>>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <[hidden email]>
>>>> wrote:
>>>>
>>>>> Oh and one more thing!
>>>>>
>>>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>>>> resolutions, which people can manually bump up to when watching videos with
>>>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>>>> only a small fraction of input files are 4K so should not significantly
>>>>> increase disk space projections for now.
>>>>>
>>>>> These can take a long time to compress, so if we find it's problematic
>>>>> we'll turn them back off until the jobs can be split into tiny chunks
>>>>> (future work planned!), but it works in my testing and shouldn't clog the
>>>>> servers now that we have more available.
>>>>>
>>>>> (Note that the ogv.js player shim for Safari will not handle
>>>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>>>> Chromium-based browser.)
>>>>>
>>>>> -- brion
>>>>>
>>>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>>>>> quality when needed, and pulling in some other improvements to the config
>>>>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>>>>
>>>>>> If all goes well with the general deployments in the next few days,
>>>>>> expect the beginning of VP9 rollout starting next week.
>>>>>>
>>>>>> Changes since the earlier announcement:
>>>>>> * the new row-multithreading will be available, which allows higher
>>>>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>>>>> slower instead of 3-4x slower.
>>>>>> * switch to constrained quality with a larger max bitrate: many files
>>>>>> will become significantly smaller in their VP9 versions, but some will
>>>>>> actually increase in exchange for a huge increase in quality -- this is
>>>>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>>>>> didn't compress well at the default low data rates.
>>>>>>
>>>>>> -- brion
>>>>>>
>>>>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Awesome sauce. Thanks Moritz!
>>>>>>>
>>>>>>> -- brion
>>>>>>>
>>>>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>>>>> > Current state on this:
>>>>>>>> >
>>>>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>>>>>>>> start with
>>>>>>>> > best performance; Moritz made a start on libvpx but we still have
>>>>>>>> to
>>>>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all
>>>>>>>> the way to
>>>>>>>> > 3.4)
>>>>>>>>
>>>>>>>> I've completed this today. We now have a separate repository
>>>>>>>> component
>>>>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>>>>>>>> 3.2.10
>>>>>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>>>>>> Debian
>>>>>>>> with a local rebuild) with backported row-mt support and linked
>>>>>>>> against
>>>>>>>> libvpx 1.7.0.
>>>>>>>>
>>>>>>>> I tested re-encoding
>>>>>>>>
>>>>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which
>>>>>>>> results in
>>>>>>>> a size reduction from 48M to 31M.
>>>>>>>>
>>>>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>>>>> enabling row-mt
>>>>>>>> gives a significant performance boost; encoding time went down from
>>>>>>>> 5:31 mins
>>>>>>>> to 3:36 mins.
>>>>>>>>
>>>>>>>> All the details can be found at
>>>>>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>         Moritz
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Wikitech-l mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>>>>
>>>>>>>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l