Search This Blog

Stream hardware-encoded H.264 video from a Raspberry Pi to a web page using WebRTC

I use the wonderful 3D printing distribution OctoPi (which in turn uses OctoPrint) on a Raspberry Pi Zero to start and monitor my 3D prints. It has built-in webcam functionality, which means you can monitor your printer and see in the web interface if the print is going well or not - really nice! The problem with the webcam is that video is encoded to MJPEG (Motion JPEG), which can be displayed in the web interface <img> tag, but is a bandwidth hog. Streaming a 320x240 picture at 10fps generates about 800kB/s of traffic. The good thing is that the RPi hardware JPEG encoder is used, so CPU usage is very low (~2%). My 3D printer is in the basement and a WiFi connection from there is only possible using an external antenna and with only a really low bandwidth available.
Knowing the RPi has an H.264 hardware encoder I set out to find a solution to stream H.264 video onto the OctoPrint web interface. These instructions apply to a Debian Jessie based Raspbian / OctoPi 0.13.
But first things first:

Video / audio formats and container format

A generally misunderstood part of digital video is codecs and container formats. Video and audio codecs define how video or audio is encoded / compressed. These are formats like MPEG2, AVC/H.264, VP8/9, WMV9, etc. for video, and e.g. MP3 or AAC for audio and they generate raw data packets. These data packets are put (multiplexed or "muxed") into a container (together with other streams like subtitles etc.) to save it to a file or stream it via a network. These container formats are e.g. MPEG-TS, AVI, MP4, MKV, FLV etc.

Streaming formats and web browser video support

Streaming video systems are fragmented and often proprietary. There is streaming via HTML (a web browser requests a resource via a regular HTTP GET request), RTP/RSTP (older streaming format), RTMP (A proprietary format used by Flash), HSL (Newer streaming protocol not supported very well yet) and more. All of these system support different codecs internally.

The <video> element

Now the problem is that modern web browsers support the MP4 / H.264 video format, but the MP4 container format is as-it-is not suitable for streaming it to web browsers via the <video> element. Let that sink in - It is simply not possible. You need browser plugins or external programs. You can play a "finished" MP4 video file via the web browser, which will do a regular HTTP GET request, download, cache and play the file, but you can not STREAM every possible format.
The standard for the <video> element only defines "open" container formats to be supported in the browser: WEBM and OGG. It supports video/audio codecs like VP8, AVC/H.264, Theora, Vorbis.

Streaming Theora / OGG video to VLC with GStreamer

To stream video with with GStreamer first install the necessary packages with:

gstreamer1.0-omx gives us the GStreamer filter omxh264enc which enables H.264 hardware-encoding. gstreamer1.0-tools provides gst-launch1.0 which is used to build a GStreamer pipeline and start playing / streaming it. You can now try to start streaming in Theora / OGG format:

You can watch both streams in VLC when you open the address "tcp://RPI_IP:7272".
Note that the Theora stream creates a lower data rate stream, but a very high CPU usage. The video is very jerky, so I suspect it drops frames, because the CPU can not sustain encoding at 10fps, which in turn lowers the data rate...
The H.264-encoded video is smooth though and visually ok for the data rate.

Using WebRTC via Janus / Nginx

Now we know the RPi can hardware-encode and stream H.264 video. But how do we manage to view our video on a webpage?
The Firefox API page mentions RTP/RTSP as a source for the <video> tag, but I couldn't get that to work. After searching the internet for a while I ran across WebRTC, which is a realtime audio / video communication standard and seems to be supported in all major browsers. I decided to give it a try and followed these instructions and it worked:

Streaming H.264 using WebRTC

Note that you need to stop the haproxy and octoprint services before installing and starting nginx and janus, otherwise it won't work!

Using WebRTC in OctoPrint?

Ok. Now we know this works, but how can we integrate that into OctoPrint?! It is a bit unfortunate to use Janus and Nginx, another web server. OctoPrint already uses the Tornado web server, so we have to convert the example page from above to use that. I found this example on how to set up a WebRTC chat application and went from that...

Post a Comment

Popular Posts

Installing newer gcc/g++ versions is easy: sudo apt-get install gcc-4.7 g++-4.7
If your system does not provide the new versions, you might still be able to get them via a different repository. Add the toolchain repository to your system and update your sources:sudo add-apt-repository ppa:ubuntu-toolchain-r/testsudo apt-get update
Now you should be able to install gcc/g++. The real problem comes after that. How to make use of the newer versions?

Option #1 - Uninstall older versions
This is sometimes not an option, because lots of dependencies will be uninstalled which you might need, or you want to keep those versions around for compiling stuff that still uses them.

Option #2 - Use update-alternatives to switch versionsThis informative post describes it pretty well. I'll sum it up here. First remove all update-alternatives:sudo update-alternatives --remove-all gccsudo update-alternatives --remove-all g++
Now add update-alternatives for 4.6 and 4.7 and make the g++ configuration a s…

Wow. This was a royal pita. I wanted to build this wake-up-light-alarm-clock-media-thingy for my girlfriend. Well, I got my hands on a RPi Zero and needed an LCD display. I found this nice-looking "WaveShare 4" IPS display" on Ebay and spent days trying to get it to work properly. You want a hint? DONT DO IT! BUY A HDMI SCREEN! Refresh rate sucks, Kodi is not working with hardware-acceleration, Framebuffer copying never worked. Short: It sucked!

So. Next chapter: Bought a 5" HDMI screen here. It has an HDMI input and connects to the RPi GPIO pins for power and touch functionality... and works ALMOST out of the box on a fresh Raspbian install (from 5/2016). Nice!

Sometimes you are in the need to read back the OpenGL backbuffer, or other framebuffers. In my case the question was how to read back a downsampled framebuffer resp. its texture to system memory. There are different methods for this and I wrote a small benchmark to test them on various systems.

UPDATE #1: A thing I've totally overlooked is that you can actually blit the backbuffer to the downsampled framebuffer directly, saving you the FBO overhead (memory, setup, rendering time). I've updated the page and the code reflecting that.UPDATE #2: The code now works on Windows and Linux (via GLX). Maybe I'll port it to GLES2 for the Raspberry Pi too... :)UPDATE #3: The code works on the Raspberry Pi now using OpenGL ES 2.0. Reading the framebuffer is quite slow though even with an overclocked device. It's not the actual reading that is slow, but that we have to render to another framebuffer first and back to the screen. Atm all buffers and textures use RGBA8888, which might s…