Video output changing to WebM VP9/Opus soon

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Video output changing to WebM VP9/Opus soon

Brion Vibber-4
In the next couple weeks I'm planning to start switching our video
transcode output from WebM VP8/Vorbis to the newer WebM VP9/Opus profile,
which saves us about 38% on file size and bandwidth while retaining the
same quality.

This will not affect what kinds of files you upload; only the scaled
transcoded output files used for playback will change. All modern browsers
that support VP8 support VP9 as well, and our player shim for Safari and IE
will continue to work.

All the details:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition

Comments and questions welcome!

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brian Wolff
Woo!

Good to see us moving forward to newer codecs.

What is the expected impact on transcode time/memory usage? (As in, will
large videos still transcode fine or would they potentially hit limits
sooner?)

--
brian

On Sunday, June 17, 2018, Brion Vibber <[hidden email]> wrote:
> In the next couple weeks I'm planning to start switching our video
transcode output from WebM VP8/Vorbis to the newer WebM VP9/Opus profile,
which saves us about 38% on file size and bandwidth while retaining the
same quality.
> This will not affect what kinds of files you upload; only the scaled
transcoded output files used for playback will change. All modern browsers
that support VP8 support VP9 as well, and our player shim for Safari and IE
will continue to work.
> All the details:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition
> Comments and questions welcome!
> -- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff <[hidden email]> wrote:

> Woo!
>
> Good to see us moving forward to newer codecs.
>

:D


> What is the expected impact on transcode time/memory usage? (As in, will
> large videos still transcode fine or would they potentially hit limits
> sooner?)
>

Using the same CPU thread count, typical files take about 3-4 times longer
with the configuration we've got ready. This doesn't seem to be a major
problem at low resolutions, but higher resolutions may need to have the
limits raised for very long videos like conference talks. Memory usage may
be higher but I've not specifically noticed a major difference between VP8
and VP9 there.

This could be improved significantly by updating to libvpx 1.7 and a
matching version of ffmpeg that supports macroblock row multithreading:
this means that we'll be able to use more than 4 threads for HD videos,
which should allow using all cores on a 20-core/40-thread machine if it's
not loaded up with other files.

The necessary libraries are released; we just need to backport the newer
packages to Debian 9 I believe. Don't yet know whether this will be an easy
or hard task.

Now that you mention it I am concerned about the time limit on very long
videos, so I'll take another look at the packaging backport and see if we
can knock that out quickly. :)

-- brion
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Current state on this:

* delaying to mid-July so I don't start a batch process and leave it
unattended for a week :)
* still hoping to deploy the libvpx+ffmpeg backport first so we start with
best performance; Moritz made a start on libvpx but we still have to
resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to
3.4)
* after further testing of 30fps and 60fps files, I'm also planning to
increase the data rate a bit.

I want to do a little more testing on the ideal data rate, but it'll end up
looking more like a 30% reduction than a 38% reduction in file size. Our
current VP8 and provisional-VP9 configurations were tuned mostly on 24fps
files, while 30fps files are very common and require a moderately higher
data rate to reduce compression artifacts. (50fps and 60fps files are also
around, and will also benefit from an increase in data rate.)

(Possibly the data rate will end up scaled according to frame rate, but
frame rate is surprisingly hard to calculate reliably for WebM input.)

-- brion


On Sun, Jun 17, 2018 at 1:32 PM Brion Vibber <[hidden email]> wrote:

> On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff <[hidden email]> wrote:
>
>> Woo!
>>
>> Good to see us moving forward to newer codecs.
>>
>
> :D
>
>
>> What is the expected impact on transcode time/memory usage? (As in, will
>> large videos still transcode fine or would they potentially hit limits
>> sooner?)
>>
>
> Using the same CPU thread count, typical files take about 3-4 times longer
> with the configuration we've got ready. This doesn't seem to be a major
> problem at low resolutions, but higher resolutions may need to have the
> limits raised for very long videos like conference talks. Memory usage may
> be higher but I've not specifically noticed a major difference between VP8
> and VP9 there.
>
> This could be improved significantly by updating to libvpx 1.7 and a
> matching version of ffmpeg that supports macroblock row multithreading:
> this means that we'll be able to use more than 4 threads for HD videos,
> which should allow using all cores on a 20-core/40-thread machine if it's
> not loaded up with other files.
>
> The necessary libraries are released; we just need to backport the newer
> packages to Debian 9 I believe. Don't yet know whether this will be an easy
> or hard task.
>
> Now that you mention it I am concerned about the time limit on very long
> videos, so I'll take another look at the packaging backport and see if we
> can knock that out quickly. :)
>
> -- brion
>
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Pine W
Thanks for the update. As you can imagine, I am very interested in this!

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

-------- Original message --------From: Brion Vibber <[hidden email]> Date: 6/28/18  1:54 PM  (GMT-08:00) To: Wikimedia Foundation Multimedia Team <[hidden email]> Cc: Wikimedia-tech list <[hidden email]> Subject: Re: [Multimedia] Video output changing to WebM VP9/Opus soon
Current state on this:
* delaying to mid-July so I don't start a batch process and leave it unattended for a week :)* still hoping to deploy the libvpx+ffmpeg backport first so we start with best performance; Moritz made a start on libvpx but we still have to resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to 3.4)* after further testing of 30fps and 60fps files, I'm also planning to increase the data rate a bit.
I want to do a little more testing on the ideal data rate, but it'll end up looking more like a 30% reduction than a 38% reduction in file size. Our current VP8 and provisional-VP9 configurations were tuned mostly on 24fps files, while 30fps files are very common and require a moderately higher data rate to reduce compression artifacts. (50fps and 60fps files are also around, and will also benefit from an increase in data rate.)
(Possibly the data rate will end up scaled according to frame rate, but frame rate is surprisingly hard to calculate reliably for WebM input.)
-- brion

On Sun, Jun 17, 2018 at 1:32 PM Brion Vibber <[hidden email]> wrote:
On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff <[hidden email]> wrote:
Woo!

Good to see us moving forward to newer codecs.

:D What is the expected impact on transcode time/memory usage? (As in, will large videos still transcode fine or would they potentially hit limits sooner?)

Using the same CPU thread count, typical files take about 3-4 times longer with the configuration we've got ready. This doesn't seem to be a major problem at low resolutions, but higher resolutions may need to have the limits raised for very long videos like conference talks. Memory usage may be higher but I've not specifically noticed a major difference between VP8 and VP9 there.
This could be improved significantly by updating to libvpx 1.7 and a matching version of ffmpeg that supports macroblock row multithreading: this means that we'll be able to use more than 4 threads for HD videos, which should allow using all cores on a 20-core/40-thread machine if it's not loaded up with other files.
The necessary libraries are released; we just need to backport the newer packages to Debian 9 I believe. Don't yet know whether this will be an easy or hard task.
Now that you mention it I am concerned about the time limit on very long videos, so I'll take another look at the packaging backport and see if we can knock that out quickly. :)
-- brion

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Moritz Muehlenhoff
In reply to this post by Brion Vibber-4
Hi all,

On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> Current state on this:
>
> * still hoping to deploy the libvpx+ffmpeg backport first so we start with
> best performance; Moritz made a start on libvpx but we still have to
> resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to
> 3.4)

I've completed this today. We now have a separate repository component
for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
(thus allowing us to follow the ffmpeg security updates released in Debian
with a local rebuild) with backported row-mt support and linked against
libvpx 1.7.0.

I tested re-encoding
https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
(which is a nice fast-paced test file) from VP8 to VP9, which results in
a size reduction from 48M to 31M.

When using eight CPU cores on one of our video scaler servers, enabling row-mt
gives a significant performance boost; encoding time went down from 5:31 mins
to 3:36 mins.

All the details can be found at
https://phabricator.wikimedia.org/T190333#4324995

Cheers,
        Moritz

_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Reply | Threaded
Open this post in threaded view
|

Re: [Multimedia] Video output changing to WebM VP9/Opus soon

Brion Vibber-4
Awesome sauce. Thanks Moritz!

-- brion

On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
[hidden email]> wrote:

> Hi all,
>
> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> > Current state on this:
> >
> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
> with
> > best performance; Moritz made a start on libvpx but we still have to
> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the way
> to
> > 3.4)
>
> I've completed this today. We now have a separate repository component
> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
> (thus allowing us to follow the ffmpeg security updates released in Debian
> with a local rebuild) with backported row-mt support and linked against
> libvpx 1.7.0.
>
> I tested re-encoding
>
> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
> (which is a nice fast-paced test file) from VP8 to VP9, which results in
> a size reduction from 48M to 31M.
>
> When using eight CPU cores on one of our video scaler servers, enabling
> row-mt
> gives a significant performance boost; encoding time went down from 5:31
> mins
> to 3:36 mins.
>
> All the details can be found at
> https://phabricator.wikimedia.org/T190333#4324995
>
> Cheers,
>         Moritz
>
> _______________________________________________
> Wikitech-l mailing list
> [hidden email]
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
[hidden email]
https://lists.wikimedia.org/mailman/listinfo/wikitech-l