VP8/VP9 RTP library

manpreet Best Answer 2 years ago

Are there any existing libraries facilitating the transport of a VP8/VP9 stream through RTP ?

I found the RFC draft for VP8 (http://tools.ietf.org/html/draft-ietf-payload-vp8-08) but nothing mentioning the VP9 codec (too recent ?). It seems that libavformat implements this draft. Does anyone have other alternatives ?

manpreet 2 years ago

As of GStreamer 1.8.2 (possibly older, but 1.8.2 or later will make your life a lot easier), this can all be done using GStreamer itself. You should IGNORE the ffmpeg and libav (same thing-- project had a schism) cross-plugs.

You will want the source for gstreamer, gst-plugins-base, gst-plugins-good, and gst-plugins-bad, that's it. Do not use gst-plugins-ffmpeg, libav, etc., all of that advice is old at this point. (You might still need it for h.264/h.265, but that's another mess.) You should also get the sources for libvpx, preferably version 1.6.0. Get good and familiar with building these projects from automake. GStreamer in particular has remarkably clean configure scripts; learn how to use them effectively.

It is important not to mix-and-match 0.10 series and 1.8.2 series code, however, as the RFCs for VP8 were ratified in the meantime (as they were for Opus, if you're using that for audio, which I recommend), and some of the magic strings that are used, for example, in SDP files have changed.

You should, however, be aware that there are still a few loose ends.

In particular, if you use SDP files to drive all this stuff, be aware that automatic graph generation using "sdpdemux" still has some limitations. For one thing, the stock sdpdemux has no "rank", and therefore will not automatically handle type "application/sdp". You should expect to build a custom application to build your graph-- and that process has a bit of a learning curve.

You should be open to branching copies of the plugins and tweaking them, as that will often add a lot of flexibility for not much effort, but be aware that copied plugins need lots of token name changes that must be done to avoid conflicts with existing plugins, and must be done with care to ensure the plugin still works.

I recommend you thoroughly familiarize yourself with the following GStreamer components:

vp8dec vp8enc vp8parse (NOTE: SEE BELOW) matroskamux matroskademux queue rtpbin rtpjitterbuffer sdpdemux udpsink udpsrc videoconvert

...and matching materials for at least one modern audio codec (I recommend Opus, so, "opusdec" and "opusenc").

Note "vp8parse" above. This is available from the "kotaku" project, and you may or may not need it. If you want to handle vp8 video at a granular level-- skipping around in it without actually decoding it, etc.-- you may need it. Certain types of non-linear editing applications will definitely need this. As of GST 1.8.2 it had not yet been merged into the mainline GStreamer codebase.

As a particular point of sanity preservation, DO NOT USE VORBIS FOR AUDIO STREAMING VIA RTP without understanding what you are getting into. Vorbis and Theora use "dynamic codebooks"-- content-dependent compression dictionaries that vary with the material being compressed. This means the decoder magically needs to know what the encoder came up with as the codebook, but you don't have a file header, because you're streaming. There are ways to work around this, but they are not pleasant. If you go down the path of trying to jam this into an SDP file for recovery at the remote end, I suggest you identify a low-cost scotch whiskey you can tolerate in quantity.

There are a few other gotchas to be wary of.

DO NOT use any of the "fancy" compression settings if you expect to be decoded on hardware-accelerated embedded systems unless you have VERY complete QA resources at your disposal. Not all technically-valid streams will decode in all hardware-accelerated decoders. In particular, the "resiliency" options are very tempting, and very effective, but will break accelerated decoding on IMX6 targets.

VP8 encoding cannot be doing in real-time without very significant compromises. It can take some time to realize what these are and how they work. Going into that is outside the scope of this question, except for the issue that you can easily think your transmitter is broken when you're actually just not getting data to your udpsinks fast enough. Don't worry, it can be done, but you need to do your research. Start by looking at the "deadline" option for libvpx.

Finally, I strongly encourage you to look at the RFCs for VP8, RTP, VP8 payloads in RTP specifically, RTCP (R-T-C-P, not RTP, not TCP), SDP, VP8 streams in SDP specifically. Each of those specific issues has several RFCs. BE SURE YOU ARE LOOKING AT THE CURRENT ONES, NOT THE DRAFTS! There have been some VERY significant changes.

As you familiarize yourself with your specs and your tools, you will realize three points that I will summarize for you now:

1. You can't do everything with gst-launch, but if you can, it saves a huge amount of time.

2. You CAN do everything with a custom application, but the learning curve is significant, and development time will also be significant.

3. There are a lot of people writing official RFCs that have only tested surprisingly narrow use cases of the material they're writing about, ESPECIALLY in multimedia.

Good luck!