This “How-to” outlines some of the issues associated with video capture on the BeagleBone Black (BBB) for robotic vision
applications using a USB webcam and OpenCV, and presents a possible solution for improved video capture performance along with a
set of detailed instructions. In particular, this document addresses the challenge of achieving a suitably high framerate (30 fps) for
robotic vision applications making use of the BBB. If you wish, you can skip the introductory material and jump directly to the
How-to.

Problem Background

Many robotic vision applications require video capture and frame processing to occur at a high rate of speed. For
example, my master’s thesis seeks to implement autonomous close formation flight of two remote controlled aircraft
using computer vision for localization. In short, the lead aircraft will be outfitted with very high intensity LEDs on
each of its wingtips and tail surfaces while the following aircraft will be equipped with a vision system consisting of
the BeagleBone Black and a USB webcam. Both vehicles will be controlled using the open source ArduPilot Mega
autopilot. The OpenCV vision software running on the BBB will detect the LEDs in each video frame, estimate
the 3-D relative position and orientation of the leader, and pass this information over to the autopilot which will
handle the control. This type of application requires high bandwidth localization estimates for the follower to maintain
its position behind the leader. The rate at which video frames can be processed and captured becomes even more
important considering the fact that the LEDs may not be detected in every frame due to background noise in the
image.

First Attempts with the PlayStation 3 Eye

For me, a reasonable place to start was by using the PlayStation 3 Eye USB webcam in combination with OpenCV. The PS3 Eye
is used by many “do-it-yourself” robotics enthusiasts due to its availability, very low cost, high framerate (up to
120 fps at 320 × 240 and up to 60 fps at 640 × 480 resolution), and multi-platform support by 3rd party drivers. [1]
Unfortunately for those who want to use the PS3 Eye with the BBB, this kind of performance can’t be expected—at least not
easily.

If you were to set up a simple OpenCV video capture program and attempted to operate the PS3 eye at 640 × 480 resolution at
60 fps, you would end up with repeated “select timeout” errors and no video frames. You would have identical results at 30 and
15 fps as well, however if you settle for 320 × 240 resolution, you would get a stream of video, but always at 30 fps regardless of the
framerate you set. Why? It turns out that the OpenCV functions for setting the video framerate do not work (at least for
Video4Linux devices) and the default 30 fps is used. In order to set the camera framerate, you have to write your own capture code
using the Video4Linux2 API. [2] But even after using custom capture code, you would find that you can only acquire
640 × 480 frames with the camera set to 15 fps. And even then you would actually be capturing frames at about 7 fps at
best.

After more days…weeks…months than I’m willing to admit, I finally came to find that issue lies with the way the PS3 Eye transfers
the frames over USB. The PS3 Eye captures video in the uncompressed YUYV pixel format and transfers the frames in bulk mode,
which guarantees data transmission but has no guarantee of latency. In the case of the BBB, the large amount of
data being sent by the webcam saturates the bulk allotment on the BBB’s USB bandwidth and select timeout errors
result. [3]

Shift to the Logitech C920 and MJPEG Format

In order to reduce the USB bandwidth required by the webcam, a compressed pixel format such as MJPEG or H.264 can be used. The
PS3 Eye does not support video compression, so I looked to the Logitech C920 USB webcam instead. H.264 compression comes
at a cost however, and will set you back about $72 to purchase a C920 on Amazon. (Other cameras, such as the
Logitech C270 also support the MJPEG pixel format, which I ended up using over H.264. However, using the Logitech
C270 will require a little extra work. The C270 does not include the Huffman table as part of the MJPEG stream
and may need to be added for the video frames to be correctly read by OpenCV and other programs. See [3] for
more.)

Since the C920 transfers compressed images in isochronos mode, it can easily deliver 640 × 480 frames at 30 fps using very little
CPU. [3] If you save the video frames to individual JPEG image files, you can easily transfer them to your desktop computer and
view them in any image viewer. If we want to use the MJPEG stream in a vision application written with OpenCV though, these
images will have to be decompressed in real-time and converted to a cv::Mat object so that OpenCV can work with the
image.

JPEG Decompression with OpenCV

Luckily, OpenCV includes functions for decoding images from a buffer, specifically the cvDecodeImage() and imdecode() functions,
depending on if you are working in C or C++. [4][5] The primary reason for using MJPEG over the H.264 compression format is
that MJPEG uses intraframe compression whereas H.264 uses interframe compression. Put simply, each MJPEG frame gets
compressed individually as a JPEG image and each compressed frame is independent of all others. H.264 on the other hand, uses
interframe prediction to “take advantage from temporal redundancy between neighboring frames to achieve higher compression
rates”. [6] While this is good for compressing video streams meant to be viewed as a continuous stream, it is not well-suited for
embedded vision applications since H.264 decompression is CPU intensive and can exhibit decompression artifacts and lag when there
is a lot of motion in the video.

If you installed OpenCV on your BBB with a package manager such as Ubuntu’s Advanced Packaging Tool (apt-get) or
Ångstrom’s opkg, odds are that you will still only see about 10-20 fps when you try and capture at 640 × 480 resolution at the 30 fps
setting. And if you profile your code by including calls to time() from the <ctime> header file, you will see that most of the
time spent by your program is dedicated to decoding the image and converting it to the cv::Mat object. Moreover,
the decompression eats up nearly all of the CPU. Luckily, there are some steps that you can take to significantly
reduce decompression time and CPU usage—leaving more time and resources for your vision program to process the
frames.

How-To: Achieve 30 fps

Disclaimer

Please keep in mind that I am not an expert in embedded Linux, OpenCV, or C/C++ programming. I am a graduate student
studying aerospace engineering. I have only been working with embedded hardware, Linux, OpenCV, and C/C++ for about a year.
My thesis has taken me on a detour into investigating ways to improve the framerate when using a USB webcam with the BeagleBone
Black. This “How-To” is essentially a compilation of other resources and an outline of the steps that I used to solve this particular
problem—your mileage may vary. As always, it is your responsibility to understand the commands you are invoking. I have spent a lot
of time on this problem and have relied heavily on the help of the open source community. I have put this guide
together as a way of giving back to the open source community, and hope that some can find it useful. If you have any
comments or suggestions for improving this “How-To” please email me at the address provided at the top of this
page.

Prerequisites

This process can be followed with some slight variations to your setup. For reference, here is a list of what hardware and software I
used.

Before attempting any of these steps, you should already be familiar with the basics of using ssh to connect to the BBB over USB or a
LAN network. You should also be comfortable working from the Linux command line and have some experience with GNU compilers,
cmake, and the “configure, make, make install” build processes.

Objective

The main objective of this How-to is to take advantage of NEON hardware acceleration available on the BBB. [8] The details of how
NEON acceleration works are a bit over my head, but its usefulness is obvious: “NEON technology can accelerate multimedia and
signal processing algorithms such as video encode/decode, 2D/3D graphics, gaming, audio and speech processing, image processing,
telephony, and sound synthesis by at least 3x the performance of ARMv5 and at least 2x the performance of ARMv6 SIMD.” You can
also see the clear benefits here. [16]

In order to use NEON, we will (1) build and install a more optimized JPEG codec called “libjpeg-turbo”, and (2) rebuild OpenCV
with NEON enabled. The latter is a bit tricky due to the limited processing power and storage capacity of the BBB. To speed up the
build, I will introduce an easy way to cross-compile large projects for the BBB.

1. Install libjpeg-turbo

libjpeg-turbo is a highly-optimized version of the libjpeg JPEG codec library that is designed to take advantage of NEON
acceleration. According to the libjpeg-turbo project page, the library is capable of encoding/decoding JPEG images 2–4 times faster
than the standard libjpeg library. [9]

By default libjpeg-turbo will install into /opt/libjpeg-turbo. You may install to a different directory
by passing the --prefix option to the configure script. However, the remainder of these instructions
will assume that libjpeg-turbo was installed in its default location.../configure CPPFLAGS=’-O3 -pipe -fPIC -mfpu=neon -mfloat-abi=hard’makesudo make install

Note that the -O3, -fPIC, and -mfpu=neon are particularly important as they enable code optimization, position-independent code
generation, and NEON hardware acceleration, respectively.

2. Setting up for Distributed Cross-Compilation (Optional, but Recommended)

Since OpenCV is such a large project and the BBB has limited processing power, it is much more convenient to set up
cross-compilation. Typically, setting up a cross-compilation environment can be a tedious process and is especially
cumbersome when building a large project that has many dependencies which, in turn, depend on other dependencies,
etc…etc.

Fortunately with distributed cross compiling, you can take advantage of the libraries already installed on the BBB while still using
your (probably x86) PC to cross-compile the object files much faster than if they were compiled locally. With distributed
cross-compilation you can execute the build from the BBB just as if you were building on the BBB, itself. Here is how you can set up
a distributed cross-compiler with distcc: [10]

Extract the files and set the cross-compiler (CC) to the one you just installedtar xzvf gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux.tar.xzexportCC=‘pwd‘/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux/bin/arm-linux-gnueabihf-

Confirm that the correct cross-compiler is active.${CC}gcc --version

>> arm-linux-gnueabihf-gcc (crosstool-NG linaro-1.13.1-4.8-2013.08 - LinaroGCC 2013.08) 4.8.2 20130805 (prerelease)>> Copyright (C) 2013 Free Software Foundation, Inc.>> This is free software; see the source for copying conditions. There is NO>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULARPURPOSE.

Next, we will test that the compiler works by cross-compiling a simple test program for the BBB.
Create a file called hello_world.c and paste into it the following:int main(void){printf("Hello, cross-compilation world !\n");return 0;}

Compile the program with:${CC}gcc hello_world.c -o hello_world

Try running the program on your PC and confirm that it does not run../hello_world

You should be returned an error similar to the following:bash: ./hello_world: cannot execute binary file

Now copy the executable to the BBB using scp. Your command should be similar to:scp hello_world ubuntu@192.168.7.2:~/hello_world

Then execute the program on the BBB. You should see the “Hello, cross-compilation world !” message
written to stdout.cd~./hello_world>> Hello, cross-compilation world !

If the above example of cross-compilation worked, you can now move on to setting up for distributed cross-compilation. We will
begin by installing distcc on both the PC and the BBB. According to [11], we should build the most recent version of distcc from
source to take advantage of a few features.

On both the PC and the BBB, begin by installing some prerequisite packages.sudo apt-get install subversion autoconf automake python python-dev binutils-devlibgtk2.0-dev

Now that distcc is installed, we need to create some symbolic links on the BBB. Be careful to use
the correct paths here or else you may overwrite one or more of your GNU compilers. First, check which gcc and distcc are being called by defaultwhich gcc>> /usr/bin/gccwhich distcc>> /usr/local/bin/distcc

Now check your PATH variable to see if /usr/local/bin is included before /usr/bin. If it is not, then
prepend /usr/local/bin to your PATH. For example:echo $PATH>> /usr/sbin:/usr/bin:/sbin:/binexport PATH=/usr/local/bin:$PATH

So now you can check distcc is being called correctly (through the symlink you just created) by
checking the following:which gcc/usr/local/bin/gcc

If this doesn’t check out, then go back and make sure that you have all the symbolic links correct and
prepended /usr/local/bin to your PATH variable. It is important that you add to the beginning of
the path since gcc/cc/g++/c++ get called in the order they appear on the path.

Now we will create some environment variables for distcc in order to control some settings. I will
just introduce the commands here, but if you want to read more, refer to [11] and [12]. You will
need to have a working network (Either over LAN or USB through which the BBB and your PC can
communicate.) The first environment variable sets the IP address of your PC that will be doing the
compilation followed by a forward slash and the “number of jobs per machine”. A good rule of thumb
is to use twice your number of processor cores. So for me, I would use:export DISTCC_HOSTS="192.168.2.3/4"

For the rest of the environment variables use the following:export DISTCC_BACKOFF_PERIOD=0export DISTCC_IO_TIMEOUT=3000export DISTCC_SKIP_LOCAL_RETRY=1

In my case, I found that I had to execute the following command on the BBB before trying to build
OpenCV so that the cmake build process would use gcc rather than cc, which I did not have installed:export CC=/usr/local/bin/gcc

Now, we have to move back to the PC and create some similar symlinks.cd gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux/binln -s arm-linux-gnueabihf-gcc gccln -s arm-linux-gnueabihf-cc ccln -s arm-linux-gnueabihf-g++ g++ln -s arm-linux-gnueabihf-c++ c++ln -s arm-linux-gnueabihf-cpp cpp

You will have to prepend to your path as well to make the cross-compiler active.export PATH=$HOME/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux/bin:$PATHwhich gcc>> /home/uname/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux/bin/gcc

When you are about ready to start compiling OpenCV, you can launch the distcc daemon with
the following command (Your command may be different depending on the number of jobs, and your
BBB’s IP address):distccd --daemon --jobs 4 --allow 192.168.2.9 --verbose --log-stderr--no-detach

At this point, you could begin building/compiling any program or project on the BBB and you should see some activity on your
PC as it receives the jobs from the BBB and compiles the object files. In this case, you probably want to build OpenCV—for which
you should go through the next section. But once you are done building, you will most likely want to be able to disable
distcc so that you can compile programs natively on your BBB and PC again. Here is how you can disable distcc.

On the BBB, disable all of the symlinks you created.sudo rm /usr/local/bin/

The easiest way to restore all of your environment variables is simply to restart the BBB.sudo reboot

Now you will want to confirm that gcc is called instead of distcc. You can use the following commands.
You may also want to try compiling a small hello_world.c example.which gcc>> /usr/bin/gcc

If you end up with errors that the gcc compiler doesn’t exist, then you can remove then reinstall the
compilers with:sudo apt-get remove --purge build-essentialsudo apt-get install build-essential

Then we will do similarly on the PC. Make sure you are in the directory that you installed the
cross-compiler: gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux/binsudo rm gcc cc cpp g++ c++

Then reboot your computer to restore the PATH. Again, test to confirm that you can compile programs
locally on your PC.

3. Building OpenCV with libjpeg-turbo and NEON

Now we will build OpenCV with libjpeg-turbo as the JPEG codec and with NEON hardware acceleration enabled. You do not have to
use distributed cross-compiling, as described above, but it will dramatically reduce the build time. Due to the limited storage on the
BBB, you will probably need some kind of external storage. I actually used a 16 GB μSD card and a USB card reader, but a regular
USB thumb drive will probably work fine.

The first thing we have to do is reformat the USB drive with an ext2 partition. You might be able to use another kind of
filesystem, but it needs to support symbolic links—which ext2 does. Depending on your OS this process will be different, but if your
are running Ubuntu insert the USB drive and start up GParted. [13] (If GParted is not installed, you can install it with sudoapt-get install gparted.) In the top right corner select your USB drive. Please be certain that you have chosen the correctdevice, or else you could end up erasing data on your PC’s hard drive. Go to Device > Create Partition Table and
accept the prompt. Then to create the ext2 partition, right click on the “unallocated space” partition and in the File
System drop-down menu, select ext2 and “Add” the partition. When complete, eject the USB drive and remove it.

From the BBB’s terminal we need to mount the device. To know which drive is the USB drive, use
the following command, insert the USB drive, and repeat the command. The new drive is the thumb
drive. I will assume this /dev/sda with one partition, /dev/sda1 as it was for me.ls /dev/sd*

I found that it was easiest to do the following commands as root. (Be careful!)sudo su

Enter the directory that is created—whatever it is called. Then create a build directory and enter
that.cd OpenCVmkdir releasecd release

Now we will run cmake with a bunch of flags that should enable libjpeg-turbo and NEON. note that
some of these flags, such as USE_VFPV3=ON and USE_NEON=ON may have no effect, as they only work
when cross-compiling without distcc[15]. That is okay—the -mfpu=neon flag will enable NEON for
us. I’ve just gone ahead and included all of the flags that I used anyways. If you are using distcc to
cross-compile, make sure that you see some activity in your PC’s terminal after you execute the cmake
command. (The PC should compile a few programs while cmake tests the compilers it has available.)cmake -D CMAKE_C_FLAGS=’-O3 -mfpu=neon -mfloat-abi=hard’ -D CMAKE_CXX_FLAGS=’-O3-mfpu=neon -mfloat-abi=hard’ -D CMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local -D BUILD_PYTHON_SUPPORT=ON -DWITH_JPEG=ON-DBUILD_JPEG=OFF -DJPEG_INCLUDE_DIR=/opt/libjpeg-turbo/include/-DJPEG_LIBRARY=/opt/libjpeg-turbo/lib/libjpeg.a -DUSE_VFPV3=ON -DUSE_NEON=ON ..

Take careful note of the information displayed to stdout after cmake runs. You will want to make sure
that FFMPEG support is enabled, -mfpu=neon appears in the release build flags, and that the JPEG
codec is libjpeg-turbo. If everything looks okay, go ahead and begin the build…Even with distcc, the
build will take awhile.make

Now you can install the files to their default location, /usr/local and exit as the root user.make installexit

At this point, you will want to make sure that you can compile a simple OpenCV program. You can use
the framegrabberCV.c program used in the testing section. I did this by navigating to the directory
where I saved the program and executing: (See the Testing section)gcc framegrabberCV.c -o framegrabberCV ‘pkg-config --cflags --opencv opencvlibv4l2‘ -lm

If the program compiles successfully, its likely that everything installed correctly. We can now unmount
the USB drive. (NOTE: I had some issues with the USB drive being corrupted after removing it from
the BBB. You may want to first compress the OpenCV source directory (including build files) to a
.tar.gz and transfer it to your PC using scp.) When you are ready to remove the drive:sudo umount /dev/sda1sudo umount /mnt/ext2sudo eject /dev/sdasudo rm -rf /mnt/ext2

4. Testing

Testing the Framerate

Now it’s time to test the framerate. First, download the program framegrabberCV.c to the BBB and compile it.

Now take the the number of frames you captured and converted to OpenCV image objects (in this
case, 1000) and divide it by the “real” time provided by the time function to get the framerate.

Incorporating Custom Capture Code as an OpenCV object

If you like, you can make some changes to the custom capture code in frmegrabberCV.c and compile it as a C++ class instead of a
standalone command line program. That way you can create a capture object within your OpenCV code, grab and decode images,
and process it using OpenCV functions and methods.

Acknowledgments

I would like to give thanks to those in the open source community who have been enormously helpful. Thanks to Matthew Witherwax
for all of his help in arriving at a solution [3]; Martin Fox for providing custom Video4Linux capture code [18]; Alexander Corcoran
for answering my newbie Linux questions [19]; Robert C. Nelson for maintaining Ubuntu for BBB [7]; Max Thrun for answering
some questions about the PS3 Eye drivers [20]; and everyone else who has had the patience to help me along my
way.