November 12, 2009

The aim of this article is to describe how to use Elphel cameras with GStreamer from simple pipeline to a complex python software integration using OpenCV and other optimized video processing blocks. All the examples on this page are Free Software and are available as templates for your own application. Most of the described and linked software are available under GNU GPL, GNU LGPL and BSD-like licenses.

I demonstrate the use of the GStreamer framework with Elphel NC353L cameras series, but most of the examples can be used with (or without) any camera by replacing the RTSP source by a v4l2, dv, gnomevfs, videotest, gltest or file source.

The GStreamer framework itself, as most of the modules, is available under the GNU LGPL license, thuse it is perfectly suitable for both Free Software and proprietary product integration.About GStreamer:
GStreamer is a multimedia framework based on a pipeline concept. Designed to be cross-platform, it is known to work on GNU/Linux (x86, PowerPC and ARM), Android (OMAP3), Solaris (Intel and SPARC), Mac OS X and Microsoft Windows. GStreamer has bindings for programming-languages like Python, C++, Perl, GNU Guile and Ruby. Gstreamer is a free software, licensed under the GNU Lesser General Public License.

GStreamer is widely used by various corporations (Nokia, Motorola, Texas Instruments, Freescale, Google, and many more), Free Software & Open Source communities and has become a very powerful cross-platform multimedia framework.

GStreamer uses a plugin architecture, which makes the most of GStreamer’s functionality implemented as shared libraries. GStreamer’s base functionality contains functions for registering and to load plug-ins as well as to provide the base classes. Plugin libraries are dynamically loaded to support a wide spectrum of codecs, container formats, input/output drivers and effects.

Plug-ins can be installed semi-automatically when they are first needed. For that purpose distributions can register a back-end that resolves feature-descriptions to package-names.

Camera, PC hardware and software config:
For the scope of this article and in order to simplify GST pipelines and software dependencies we chose to limit the camera resolution to 1920×1088 (FullHD), 25 FPS with auto-exposure at max. 39ms exposure time. The configuration was made once using camvc (standard camera control interface) and recorded as default configuration using parsedit.php

The hardware I used in my tests is my daily use notebook – MacBook Pro 3.1 running 64 bits version of Ubuntu 9.04 GNU/Linux, which is not a particularly powerful machine nowadays.

Most of the video software available in Ubuntu 9.04 is compatible with Elphel cameras “out of the box”, but limited to FullHD resolution. A few software modules and libraries need to be patched to be able to fully benefit from Elphel’s high resolution.

GStreamer is a library that can be included in your own software, but there is gst-launch program to allow quick & easy prototyping. As such, you certainly do not fully benefit from the flexibility of GStreamer, but it enables to prototype easily a video processing pipeline.

Matroska provides time-coding support and enables to record variable frame rate MJPEG video. Unfortunately only gstreamer was able to play such files. But of course we can re-encode to a 90 FPS video duplicating frames when necessary, to maintain the time-line.

“-e” option in gst-launch command line force EOS on sources before shutting the pipeline down. (Useful then you write to files and want to shut down by killing gst-launch with CTRL+C or with kill command)

Elphel camera streamer by default stream over unicast, but can easily be switched to multicast. In order to do so, open the camera web interface, choose “Parameter editor”, select “Streamer settings” and switch STROP_MCAST_EN from 0 to 1.

You can optionally set the multicast address.

Recent GStreamer version will automatically detect the type of the stream, but you can also specify the protocol manually:

You also need to have a route for the multicast address. If the camera is connected to your LAN and you have a default route on this Ethernet device you will automatically be routed to the multicast address. If your default route is on another network device such as wireless network card, or if you have a complex network setup you need to declare a static route to the multicast. Let’s say your wireless is wlan0 and your network connected to the camera is on eth0:

route add -net 232.0.0.0 netmask 255.0.0.0 dev eth0

JP4 custom image format plug-in:

JP4 is Elphel in-house developed image format.

The Elphel camera internal FPGA code uses a very simple algorithm to calculate YCbCr from the Bayer pixels. It just uses 3×3 pixel blocks. The other point is that this algorithm is time consuming and with the 5MPix sensor the debayering process became a bottleneck. So we have added a special JP4 mode that bypasses the demosaicing in the FPGA and provides an image with pixels rearranged in 16×16 macro-blocks to separate each bayer-color in an individual 8×8 sub-block, to save space this image is then encoded as monochrome.

You may find out more information about different JP4 modes in Andrey Filippov‘s last Linux for devices article.

Recently a small FLOSS company – UbiCast, specialized in intuitive and integrated solutions to create and share high value video content contributed to a JP4 debayer plug-in for GStreamer. This plug-in implements a de-block and several Debayer algorithms available from libdc1394. Currently only JP46 mode is supported, but the structure is in place to create support for other modes and to try different kinds of optimizations.

It is important to note that OMAP and DaVinvi also have OpenGL ES (Embedded Standard) support.

OpenGL & GLSL

You can implement algorithms on the GPU via OpenGL and GLSL, the OpenGL shader language. OpenGL and GLSL are not limited to nVidia GPUs (G80 and newer) like CUDA is. This helps to bring your optimizations to a wider range of GPUs from different manufacturers.

Dirac/Schrödinger for example have an OpenGL and GLSL optimization for GStreamer, the project is available on Dirac GSoC 2008.

GStreamer has a very powerful GL plug-in. It allows you to transfer data between CPU and GPU and has a plugin structure for writing custom shaders for GPU based processing. You can write your own plug-ins, examples are available on the gst-plugins-gl project.

OpenCV is a computer vision library originally developed by Intel. It is available for commercial and research use under the open source BSD license. The library is cross-platform. It focuses mainly on real-time image processing and can be optimized with Intel’s Integrated Performance Primitives, DSP, FPGA or GPU optimized routines to accelerate itself.

Here is a list of plug-ins currently available with gst-opencv: edgedetect, faceblur, facedetect, pyramidsegment, templatematch. It is easy to implement any OpenCV function as gst plugin using the existing modules as reference.

setparam.php is a simple example to set sensor parameters from a PHP script.snapfull.php is a more complex example. It takes a full-frame snapshot and returns to previous resolution with minimal delay.

ElphelVision software from Apertus Open Source Cinema project demonstrates a nice integration with Elphel. Developed as Java Applet using MPlayer to display the live video stream ElphelVision communicates with custom PHP scripts running on the camera to set and read sensor/camera parameters.

Michael Niedermayer who patched libswscale to support Elphel’s high resolution video. Now MPlayer, VLC, Blender, GStreamer and other software built on top of FFMPEG can be used to rescale high resolution videos (up to 5012×5012).

Konstantin Kim for the first de-block/debayer implementation in MPlayer and AviSynth, for his matlab scripts and for the support he offered me while I was writing this paper.

UbiCast company and especially Florent Thiery and Anthony Violo for their contribution to the JP4 support for GStreamer and for their gstmanager.

Luc Deschenaux for introducing me to GStreamer; for the rtspsrc, rtpjpegdepay and gst-plugins-gl patches; the emerging glshader plugin and GLSL scripts.. And all the good rum (which is to blame if I forgot to mention something…)

Hi,
I found really interesting this post, because I’m creating an app that use opencv and elphel 353 camera

Unfortunately to decode the rtsp stream, I need to get it from camera with mencoder and AVLD (videoloopback) but these are not very stable. I wonder what is the best way to connect to the camera to get its frames for image processing (opencv)?

Esteban, do you need just individual frames for processing? Then you do not need the streamer, you can get them from the image server (imgsrv) running in the camera on port 8081, it has embedded help (displayed when you just open the http://capera_ip:8081)

AVLD and different videoloopback devices are good for prototyping or for using with proprietary software where you can not implement RTSP support. OpenCV can be now compiled with FFMPEG or GSTreamer. Both should support RTSP video. I have never tried it yet. (You need CVS version of OpenCV)
Regards,
Alexandre

My name is Vitaliy Petrakov, account manager at Opteck. My mail Vitaliy.Petrakov@yandex.ru. I’d like to know how it’s possible to purchase traffic from you? Your website is exactly the right type to represent our platform. We offer competitive CPA and Revenue Share commission plans. Our trading platform is offered in 9 different languages.